[jira] [Commented] (HIVE-19347) TestTriggersWorkloadManager tests are failing consistently
[
https://issues.apache.org/jira/browse/HIVE-19347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471608#comment-16471608
]
Hive QA commented on HIVE-19347:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922725/HIVE-19347.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 13576 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,mm_update.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,udf_sort_array_by_wrong2.q,loc
[jira] [Updated] (HIVE-19347) TestTriggersWorkloadManager tests are failing consistently
[
https://issues.apache.org/jira/browse/HIVE-19347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-19347:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> TestTriggersWorkloadManager tests are failing consistently
> --
>
> Key: HIVE-19347
> URL: https://issues.apache.org/jira/browse/HIVE-19347
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vineet Garg
>Assignee: Matt McCline
>Priority: Blocker
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19347.01.patch, HIVE-19347.02.patch
>
>
> Caused by the patch which turned on vectorization. Following tests are
> failing due to the patch:
> * org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
> *
> org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
> *
> org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
> * org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite
> 10 sec 14
> * org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps
> 7.7 sec 14
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
> 15 sec 14
> * org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
> 17 sec 14
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
> 1.5 sec 14
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
> 20 sec 18
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
> 1.4 sec 18
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
> 2.6 sec 18
> * org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
> {noformat}
> Error Message
> Expected query to succeed expected null, but was: Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 3,
> vertexId=vertex_1524884047358_0001_21_01, diagnostics=[Task failed,
> taskId=task_1524884047358_0001_21_01_00, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1524884047358_0001_21_01_00_0:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:80)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> ... 15 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19347) TestTriggersWorkloadManager tests are failing consistently
[
https://issues.apache.org/jira/browse/HIVE-19347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-19347:
Fix Version/s: 3.1.0
3.0.0
> TestTriggersWorkloadManager tests are failing consistently
> --
>
> Key: HIVE-19347
> URL: https://issues.apache.org/jira/browse/HIVE-19347
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vineet Garg
>Assignee: Matt McCline
>Priority: Blocker
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19347.01.patch, HIVE-19347.02.patch
>
>
> Caused by the patch which turned on vectorization. Following tests are
> failing due to the patch:
> * org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
> *
> org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
> *
> org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
> * org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite
> 10 sec 14
> * org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps
> 7.7 sec 14
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
> 15 sec 14
> * org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
> 17 sec 14
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
> 1.5 sec 14
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
> 20 sec 18
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
> 1.4 sec 18
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
> 2.6 sec 18
> * org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
> {noformat}
> Error Message
> Expected query to succeed expected null, but was: Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 3,
> vertexId=vertex_1524884047358_0001_21_01, diagnostics=[Task failed,
> taskId=task_1524884047358_0001_21_01_00, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1524884047358_0001_21_01_00_0:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:80)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> ... 15 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19347) TestTriggersWorkloadManager tests are failing consistently
[
https://issues.apache.org/jira/browse/HIVE-19347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471614#comment-16471614
]
Matt McCline commented on HIVE-19347:
-
Committed to master and branch-3. [~sershe] thank you for your code review!
> TestTriggersWorkloadManager tests are failing consistently
> --
>
> Key: HIVE-19347
> URL: https://issues.apache.org/jira/browse/HIVE-19347
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vineet Garg
>Assignee: Matt McCline
>Priority: Blocker
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19347.01.patch, HIVE-19347.02.patch
>
>
> Caused by the patch which turned on vectorization. Following tests are
> failing due to the patch:
> * org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
> *
> org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
> *
> org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
> * org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite
> 10 sec 14
> * org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps
> 7.7 sec 14
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
> 15 sec 14
> * org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
> 17 sec 14
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
> 1.5 sec 14
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
> 20 sec 18
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
> 1.4 sec 18
> *
> org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
> 2.6 sec 18
> * org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
> {noformat}
> Error Message
> Expected query to succeed expected null, but was: Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 3,
> vertexId=vertex_1524884047358_0001_21_01, diagnostics=[Task failed,
> taskId=task_1524884047358_0001_21_01_00, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1524884047358_0001_21_01_00_0:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:80)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> ... 15 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17852) remove support for list bucketing "stored as directories" in 3.0
[ https://issues.apache.org/jira/browse/HIVE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471620#comment-16471620 ] Laszlo Bodor commented on HIVE-17852: - Just for reference: this was introduced by HIVE-3649 > remove support for list bucketing "stored as directories" in 3.0 > > > Key: HIVE-17852 > URL: https://issues.apache.org/jira/browse/HIVE-17852 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Laszlo Bodor >Priority: Major > > From the email thread: > 1) LB, when stored as directories, adds a lot of low-level complexity to Hive > tables that has to be accounted for in many places in the code where the > files are written or modified - from FSOP to ACID/replication/export. > 2) While working on some FSOP code I noticed that some of that logic is > broken - e.g. the duplicate file removal from tasks, a pretty fundamental > correctness feature in Hive, may be broken. LB also doesn’t appear to be > compatible with e.g. regular bucketing. > 3) The feature hasn’t seen development activity in a while; it also doesn’t > appear to be used a lot. > Keeping with the theme of cleaning up “legacy” code for 3.0, I was proposing > we remove it. > (2) also suggested that, if needed, it might be easier to implement similar > functionality by adding some flexibility to partitions (which LB directories > look like anyway); that would also keep the logic on a higher level of > abstraction (split generation, partition pruning) as opposed to many > low-level places like FSOP, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19374) Parse and process ALTER TABLE SET OWNER command syntax
[
https://issues.apache.org/jira/browse/HIVE-19374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471624#comment-16471624
]
Peter Vary commented on HIVE-19374:
---
[~spena]: Thanks for your work on this!
We created a set of HMS API tests, so we can check the compatibility of the
changes, and test the expected responses for the HMS API calls in some time
back for the most important HMS API calls. You can see the work here:
[https://github.com/apache/hive/tree/master/standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client]
We would like to keep them up-to-date with every HMS API change so as mentioned
in HIVE-19371, it would be good to extend the current tests with the default
value, and the modification of the {{OwnerType}} field as well?
{{TestTablesCreateDropAlterTruncate.testCreateTableDefaultValues}}
[https://github.com/apache/hive/blob/master/standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesCreateDropAlterTruncate.java#L244]
and
{{TestTablesCreateDropAlterTruncate.testAlterTable}}
[https://github.com/apache/hive/blob/master/standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesCreateDropAlterTruncate.java#L662]
Also it would be nice to extend the TableBuilder to handle the new attribute as
well:
[https://github.com/apache/hive/blob/master/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/client/builder/TableBuilder.java]
We might address these issues in another jira, if you think that would be the
best way to handle this
> Parse and process ALTER TABLE SET OWNER command syntax
> --
>
> Key: HIVE-19374
> URL: https://issues.apache.org/jira/browse/HIVE-19374
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Major
> Attachments: HIVE-19374.1.patch, HIVE-19374.2.patch,
> HIVE-19374.3.patch
>
>
> Subtask that parses the new alter table set owner syntax and implements code
> to call HMS to change the owner of a table to a user or a role.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19476) Fix failures in TestReplicationScenariosAcidTables, TestReplicationOnHDFSEncryptedZones and TestCopyUtils
[ https://issues.apache.org/jira/browse/HIVE-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19476: Attachment: HIVE-19476.02.patch > Fix failures in TestReplicationScenariosAcidTables, > TestReplicationOnHDFSEncryptedZones and TestCopyUtils > - > > Key: HIVE-19476 > URL: https://issues.apache.org/jira/browse/HIVE-19476 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Tests, pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-19476.01.patch, HIVE-19476.02.patch > > > TestReplicationScenariosAcidTables broken due to HIVE-17657. > cc [~sershe] > TestReplicationOnHDFSEncryptedZones and TestCopyUtils are failing due to > auto-stats gathered which thrown exception irrelevant to theses tests. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19476) Fix failures in TestReplicationScenariosAcidTables, TestReplicationOnHDFSEncryptedZones and TestCopyUtils
[ https://issues.apache.org/jira/browse/HIVE-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19476: Status: Open (was: Patch Available) > Fix failures in TestReplicationScenariosAcidTables, > TestReplicationOnHDFSEncryptedZones and TestCopyUtils > - > > Key: HIVE-19476 > URL: https://issues.apache.org/jira/browse/HIVE-19476 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Tests, pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-19476.01.patch, HIVE-19476.02.patch > > > TestReplicationScenariosAcidTables broken due to HIVE-17657. > cc [~sershe] > TestReplicationOnHDFSEncryptedZones and TestCopyUtils are failing due to > auto-stats gathered which thrown exception irrelevant to theses tests. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19476) Fix failures in TestReplicationScenariosAcidTables, TestReplicationOnHDFSEncryptedZones and TestCopyUtils
[ https://issues.apache.org/jira/browse/HIVE-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19476: Status: Patch Available (was: Open) Attached 02.patch with proper config in test for hive.distcp.privileged.doAs for a test failure. > Fix failures in TestReplicationScenariosAcidTables, > TestReplicationOnHDFSEncryptedZones and TestCopyUtils > - > > Key: HIVE-19476 > URL: https://issues.apache.org/jira/browse/HIVE-19476 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Tests, pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-19476.01.patch, HIVE-19476.02.patch > > > TestReplicationScenariosAcidTables broken due to HIVE-17657. > cc [~sershe] > TestReplicationOnHDFSEncryptedZones and TestCopyUtils are failing due to > auto-stats gathered which thrown exception irrelevant to theses tests. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19456) Beeline dfs command Error: Error while processing statement: (state=,code=1)
[ https://issues.apache.org/jira/browse/HIVE-19456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chuanjie.duan reassigned HIVE-19456: Assignee: Bijoy Kumar Khandelwal > Beeline dfs command Error: Error while processing statement: (state=,code=1) > > > Key: HIVE-19456 > URL: https://issues.apache.org/jira/browse/HIVE-19456 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.2.1 >Reporter: chuanjie.duan >Assignee: Bijoy Kumar Khandelwal >Priority: Major > Attachments: HIVE-19456.0.patch > > > Reproduce steps: > # hdfs dfs -chmod 750 /tmp/aaa (user:test) > # beeline login with user test1 > # beeline "dfs -ls /tmp/aaa" > # Error: Error while processing statement: (state=,code=1) > Cause: > beeline "dfs" command would call fsshell, fsshell just err.print(errmsg) and > return "null" value when catching exception -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19479) encoded stream seek is incorrect for 0-length RGs in LLAP IO
[
https://issues.apache.org/jira/browse/HIVE-19479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471643#comment-16471643
]
Hive QA commented on HIVE-19479:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} llap-server in master has 86 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
57s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} llap-server: The patch generated 0 new + 65
unchanged - 1 fixed = 65 total (was 66) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 7 new + 328 unchanged - 0
fixed = 335 total (was 328) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10820/dev-support/hive-personality.sh
|
| git revision | master / 68b66a6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10820/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10820/yetus/patch-asflicense-problems.txt
|
| modules | C: llap-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10820/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> encoded stream seek is incorrect for 0-length RGs in LLAP IO
>
>
> Key: HIVE-19479
> URL: https://issues.apache.org/jira/browse/HIVE-19479
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19479.01.patch, HIVE-19479.patch
>
>
> The PositionProvider offset is not updated correctly and an error like this
> may happen:
> {noformat}
> Caused by: java.lang.IllegalArgumentException: Seek in LENGTH to 541 is
> outside of the data
> at
> org.apache.orc.impl.InStream$UncompressedStream.seek(InStream.java:161)
> at
> org.apache.orc.impl
[jira] [Commented] (HIVE-19479) encoded stream seek is incorrect for 0-length RGs in LLAP IO
[
https://issues.apache.org/jira/browse/HIVE-19479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471681#comment-16471681
]
Hive QA commented on HIVE-19479:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922728/HIVE-19479.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 47 failed/errored test(s), 13452 tests
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=253)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,mm_update.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_c
[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs
[
https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471709#comment-16471709
]
Hive QA commented on HIVE-18533:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} spark-client in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} spark-client: The patch generated 11 new + 27
unchanged - 7 fixed = 38 total (was 34) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} spark-client generated 2 new + 10 unchanged - 5 fixed
= 12 total (was 15) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:spark-client |
| | Class org.apache.hive.spark.client.SparkLauncherSparkClient defines
non-transient non-serializable instance field sparkLauncher In
SparkLauncherSparkClient.java:instance field sparkLauncher In
SparkLauncherSparkClient.java |
| | org.apache.spark.launcher.InProcessLauncher stored into non-transient
field SparkLauncherSparkClient.sparkLauncher At
SparkLauncherSparkClient.java:SparkLauncherSparkClient.sparkLauncher At
SparkLauncherSparkClient.java:[line 182] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10821/dev-support/hive-personality.sh
|
| git revision | master / 68b66a6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10821/yetus/diff-checkstyle-spark-client.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10821/yetus/new-findbugs-spark-client.html
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10821/yetus/patch-asflicense-problems.txt
|
| modules | C: common itests itests/qtest-spark ql spark-client U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10821/yetus.txt |
| Powered by | Apac
[jira] [Commented] (HIVE-19248) REPL LOAD couldn't copy file from source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471721#comment-16471721
]
ASF GitHub Bot commented on HIVE-19248:
---
Github user sankarh closed the pull request at:
https://github.com/apache/hive/pull/342
> REPL LOAD couldn't copy file from source CM path and also doesn't throw error
> if file copy fails.
> -
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Critical
> Labels: DR, pull-request-available, replication
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19248.01-branch-3.patch, HIVE-19248.01.patch,
> HIVE-19248.02.patch
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> Distcp failed as the CM path for the file doesn't point to source file
> system. So, it is needed to get the qualified cm root URI as part of files
> listed in dump.
> Also, REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, then retry with same set
> of files again but need to set CM path if original source file is missing or
> modified based on checksum. Let distcp to skip the properly copied files.
> FileUtil.copy will always overwrite the files.
> # If source path is moved to CM path, then delete the incorrectly copied
> files.
> # If copy fails for maximum atte
[jira] [Assigned] (HIVE-19499) Bootstrap REPL LOAD shall add tasks to create checkpoints for tables/partitions.
[ https://issues.apache.org/jira/browse/HIVE-19499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan reassigned HIVE-19499: --- > Bootstrap REPL LOAD shall add tasks to create checkpoints for > tables/partitions. > > > Key: HIVE-19499 > URL: https://issues.apache.org/jira/browse/HIVE-19499 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Fix For: 3.1.0 > > > Currently. bootstrap REPL LOAD expect the target database to be empty or not > exist to start bootstrap load. > But, this adds overhead when there is a failure in between bootstrap load and > there is no way to resume it from where it fails. So, it is needed to create > checkpoints in table/partitions to skip the completely loaded objects. > Use hash of the fully qualified path of the dump directory as a checkpoint > identifier. This should be added to the table / partition properties in hive > via a task, as the last task in the DAG for table / partition creation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19499) Bootstrap REPL LOAD shall add tasks to create checkpoints for tables/partitions.
[ https://issues.apache.org/jira/browse/HIVE-19499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19499: Labels: DR replication (was: ) > Bootstrap REPL LOAD shall add tasks to create checkpoints for > tables/partitions. > > > Key: HIVE-19499 > URL: https://issues.apache.org/jira/browse/HIVE-19499 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, replication > Fix For: 3.1.0 > > > Currently. bootstrap REPL LOAD expect the target database to be empty or not > exist to start bootstrap load. > But, this adds overhead when there is a failure in between bootstrap load and > there is no way to resume it from where it fails. So, it is needed to create > checkpoints in table/partitions to skip the completely loaded objects. > Use hash of the fully qualified path of the dump directory as a checkpoint > identifier. This should be added to the table / partition properties in hive > via a task, as the last task in the DAG for table / partition creation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs
[
https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471743#comment-16471743
]
Hive QA commented on HIVE-18533:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922715/HIVE-18533.94.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 44 failed/errored test(s), 13579 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,mm_update.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,udf_sort_array_by_wrong2.q,
[jira] [Assigned] (HIVE-13745) UDF current_date、current_timestamp、unix_timestamp NPE
[ https://issues.apache.org/jira/browse/HIVE-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen reassigned HIVE-13745: --- Assignee: Yongzhi Chen (was: Biao Wu) > UDF current_date、current_timestamp、unix_timestamp NPE > - > > Key: HIVE-13745 > URL: https://issues.apache.org/jira/browse/HIVE-13745 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Biao Wu >Assignee: Yongzhi Chen >Priority: Major > Attachments: HIVE-13745.1.patch, HIVE-13745.patch > > > NullPointerException when current_date is used in mapreduce -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-13745) UDF current_date、current_timestamp、unix_timestamp NPE
[ https://issues.apache.org/jira/browse/HIVE-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471748#comment-16471748 ] Yongzhi Chen commented on HIVE-13745: - Also attach fix for branch-2 > UDF current_date、current_timestamp、unix_timestamp NPE > - > > Key: HIVE-13745 > URL: https://issues.apache.org/jira/browse/HIVE-13745 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Biao Wu >Assignee: Yongzhi Chen >Priority: Major > Attachments: HIVE-13745.1.patch, HIVE-13745.2-brach-2.patch, > HIVE-13745.patch > > > NullPointerException when current_date is used in mapreduce -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-13745) UDF current_date、current_timestamp、unix_timestamp NPE
[ https://issues.apache.org/jira/browse/HIVE-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-13745: Attachment: HIVE-13745.2-brach-2.patch > UDF current_date、current_timestamp、unix_timestamp NPE > - > > Key: HIVE-13745 > URL: https://issues.apache.org/jira/browse/HIVE-13745 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Biao Wu >Assignee: Yongzhi Chen >Priority: Major > Attachments: HIVE-13745.1.patch, HIVE-13745.2-brach-2.patch, > HIVE-13745.patch > > > NullPointerException when current_date is used in mapreduce -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-13745) UDF current_date、current_timestamp、unix_timestamp NPE
[ https://issues.apache.org/jira/browse/HIVE-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen reassigned HIVE-13745: --- Assignee: Biao Wu (was: Yongzhi Chen) > UDF current_date、current_timestamp、unix_timestamp NPE > - > > Key: HIVE-13745 > URL: https://issues.apache.org/jira/browse/HIVE-13745 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Biao Wu >Assignee: Biao Wu >Priority: Major > Attachments: HIVE-13745.1.patch, HIVE-13745.2-brach-2.patch, > HIVE-13745.patch > > > NullPointerException when current_date is used in mapreduce -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19166) TestMiniLlapLocalCliDriver sysdb failure
[ https://issues.apache.org/jira/browse/HIVE-19166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471750#comment-16471750 ] Zoltan Haindrich commented on HIVE-19166: - it's funny that someone always rewrites sysdb.q with some different crap...first HIVE-18910 now HIVE-19448... I'll wonder if there will be 1 more change before a successfull ptest run for this ticket... the queue is just 32 today! maybe I get the an answer by monday!? :) > TestMiniLlapLocalCliDriver sysdb failure > > > Key: HIVE-19166 > URL: https://issues.apache.org/jira/browse/HIVE-19166 > Project: Hive > Issue Type: Sub-task > Components: Test >Reporter: Vineet Garg >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19166.04.patch, HIVE-19166.05.patch, > HIVE-19166.1.patch, HIVE-19166.2.patch, HIVE-19166.3.patch > > > Broken by HIVE-18715 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19166) TestMiniLlapLocalCliDriver sysdb failure
[ https://issues.apache.org/jira/browse/HIVE-19166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19166: Attachment: HIVE-19166.06.patch > TestMiniLlapLocalCliDriver sysdb failure > > > Key: HIVE-19166 > URL: https://issues.apache.org/jira/browse/HIVE-19166 > Project: Hive > Issue Type: Sub-task > Components: Test >Reporter: Vineet Garg >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19166.04.patch, HIVE-19166.05.patch, > HIVE-19166.06.patch, HIVE-19166.1.patch, HIVE-19166.2.patch, > HIVE-19166.3.patch > > > Broken by HIVE-18715 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19477) Hiveserver2 in http mode not emitting metric default.General.open_connections
[
https://issues.apache.org/jira/browse/HIVE-19477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471753#comment-16471753
]
Hive QA commented on HIVE-19477:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} service in master has 50 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} service: The patch generated 0 new + 5 unchanged - 2
fixed = 5 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10822/dev-support/hive-personality.sh
|
| git revision | master / 68b66a6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10822/yetus/patch-asflicense-problems.txt
|
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10822/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hiveserver2 in http mode not emitting metric default.General.open_connections
> -
>
> Key: HIVE-19477
> URL: https://issues.apache.org/jira/browse/HIVE-19477
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Dinesh Chitlangia
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-19477.01.patch, HIVE-19477.patch
>
>
> Instances in binary mode are emitting the metric
> _default.General.open_connections_ but the instances operating in http mode
> are not emitting this metric.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19489) Disable stats autogather for external tables
[ https://issues.apache.org/jira/browse/HIVE-19489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471762#comment-16471762 ] Zoltan Haindrich commented on HIVE-19489: - I'm not sure if we should disable it globally; but there could be an option to do that - I think it would be probably be usefull to have a table level option to prevent it from happening on specific tables. Without statistics the planner will start operating in blind: I think fs level stats are not really good; auto gathering may also collect column stats which could be very usefull during estimations. afaik auto gathering should not happen during LOAD DATA statements cc: [~ashutoshc] > Disable stats autogather for external tables > > > Key: HIVE-19489 > URL: https://issues.apache.org/jira/browse/HIVE-19489 > Project: Hive > Issue Type: Sub-task > Components: Statistics >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > Hive auto-gather of table statistics can result in incorrect generation of > stats (and the stats being marked as accurate) in the case of external tables > where the data is being written by external apps. > To avoid this issue, stats autogather will be disabled on external tables > when loading/inserting into a table with existing data, if > HIVE_DISABLE_UNSAFE_EXTERNALTABLE_OPERATIONS is enabled. In this situation, > users should rely on explicitly calling ANALYZE TABLE on their external > tables to make sure the stats are kept up-to-date. > Autogather of stats will still be allowed to occur on external tables in the > case of INSERT OVERWRITE or LOAD DATA OVERWRITE, since the existing data is > being removed and so the stats calculated on the inserted/loaded data should > be accurate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19500) Prevent multiple selectivity estimations for the same variable in conjuctions
[
https://issues.apache.org/jira/browse/HIVE-19500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich reassigned HIVE-19500:
---
> Prevent multiple selectivity estimations for the same variable in conjuctions
> -
>
> Key: HIVE-19500
> URL: https://issues.apache.org/jira/browse/HIVE-19500
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> see HIVE-19097 for problem description
> for filters like: {{(d_year in (2001,2002) and d_year = 2001)}} the current
> estimation is around {{(1/NDV)**2}} (iff column stats are available)
> this patch targets on branch-2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19468) Add Apache license to TestTxnConcatenate
[ https://issues.apache.org/jira/browse/HIVE-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471786#comment-16471786 ] Zoltan Haindrich commented on HIVE-19468: - +1 > Add Apache license to TestTxnConcatenate > > > Key: HIVE-19468 > URL: https://issues.apache.org/jira/browse/HIVE-19468 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Igor Kryvenko >Assignee: Igor Kryvenko >Priority: Major > Attachments: HIVE-19468.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-13745) UDF current_date、current_timestamp、unix_timestamp NPE
[
https://issues.apache.org/jira/browse/HIVE-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471793#comment-16471793
]
Zoltan Haindrich commented on HIVE-13745:
-
I don't think {{System.currentTimeMillis}} should be called from HiveConf...
I don't understand what positive side this could have to have it in the config;
if its ok to get it at the time the UDF is constructed.
I think it would be better to set a "pirate" property in Driver at the start of
the query execution and use that ; we already have a query timestamp for the
session Session
> UDF current_date、current_timestamp、unix_timestamp NPE
> -
>
> Key: HIVE-13745
> URL: https://issues.apache.org/jira/browse/HIVE-13745
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Biao Wu
>Assignee: Biao Wu
>Priority: Major
> Attachments: HIVE-13745.1.patch, HIVE-13745.2-brach-2.patch,
> HIVE-13745.patch
>
>
> NullPointerException when current_date is used in mapreduce
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-13745) UDF current_date、current_timestamp、unix_timestamp NPE
[ https://issues.apache.org/jira/browse/HIVE-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471800#comment-16471800 ] Zoltan Haindrich commented on HIVE-13745: - [~ychena]: [~bill] have last commented on this ticket about 2 years ago; please assign it to yourself if you are working on it, and ask for review from someone else before commiting changes. > UDF current_date、current_timestamp、unix_timestamp NPE > - > > Key: HIVE-13745 > URL: https://issues.apache.org/jira/browse/HIVE-13745 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Biao Wu >Assignee: Biao Wu >Priority: Major > Attachments: HIVE-13745.1.patch, HIVE-13745.2-brach-2.patch, > HIVE-13745.patch > > > NullPointerException when current_date is used in mapreduce -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19454) Test failure : org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1 fails with java.lang.AssertionError
[
https://issues.apache.org/jira/browse/HIVE-19454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alisha Prabhu updated HIVE-19454:
-
Attachment: HIVE-19454.2.patch
> Test failure :
> org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1 fails
> with java.lang.AssertionError
> --
>
> Key: HIVE-19454
> URL: https://issues.apache.org/jira/browse/HIVE-19454
> Project: Hive
> Issue Type: Bug
> Environment: uname -a
> Linux pts00433-vm40 3.10.0-862.el7.ppc64le #1 SMP Wed Mar 21 22:20:19 UTC
> 2018 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Alisha Prabhu
>Priority: Major
> Attachments: HIVE-19454.1.patch, HIVE-19454.2.patch
>
>
> org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion1
> refers to the same test case.
> Maven command used is : mvn -Dtest=TestTxnCommands2 test
> Error:
> {code:java}
> [INFO] Running org.apache.hadoop.hive.ql.TestTxnCommands2
> [ERROR] Tests run: 44, Failures: 1, Errors: 0, Skipped: 3, Time elapsed:
> 618.215 s <<< FAILURE! - in org.apache.hadoop.hive.ql.TestTxnCommands2
> [ERROR]
> testNonAcidToAcidConversion1(org.apache.hadoop.hive.ql.TestTxnCommands2)
> Time elapsed: 17.557 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1(TestTxnCommands2.java:499)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Failures:
> [ERROR] TestTxnCommands2.testNonAcidToAcidConversion1:499
> [INFO]
> [ERROR] Tests run: 44, Failures: 1, Errors: 0, Skipped: 3
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19477) Hiveserver2 in http mode not emitting metric default.General.open_connections
[
https://issues.apache.org/jira/browse/HIVE-19477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471816#comment-16471816
]
Hive QA commented on HIVE-19477:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922712/HIVE-19477.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 34 failed/errored test(s), 13574 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,mm_update.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,udf_sort_array_by_wrong2.q,loc
[jira] [Updated] (HIVE-19468) Add Apache license to TestTxnConcatenate
[ https://issues.apache.org/jira/browse/HIVE-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19468: Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) pushed to master. Thank you [~ikryvenko] for fixing this! > Add Apache license to TestTxnConcatenate > > > Key: HIVE-19468 > URL: https://issues.apache.org/jira/browse/HIVE-19468 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Igor Kryvenko >Assignee: Igor Kryvenko >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19468.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19449) Create standalone jar for hive streaming module
[
https://issues.apache.org/jira/browse/HIVE-19449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471829#comment-16471829
]
Hive QA commented on HIVE-19449:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922748/HIVE-19449.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10823/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10823/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10823/
Messages:
{noformat}
This message was trimmed, see log for full details
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/security/SecurityUtil.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/util/GenericOptionsParser.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-rewrite/9.3.8.v20160314/jetty-rewrite-9.3.8.v20160314.jar(org/eclipse/jetty/rewrite/handler/RedirectPatternRule.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-rewrite/9.3.8.v20160314/jetty-rewrite-9.3.8.v20160314.jar(org/eclipse/jetty/rewrite/handler/RewriteHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/Handler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/Server.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/ServerConnector.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/handler/HandlerList.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.8.v20160314/jetty-servlet-9.3.8.v20160314.jar(org/eclipse/jetty/servlet/FilterHolder.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.8.v20160314/jetty-servlet-9.3.8.v20160314.jar(org/eclipse/jetty/servlet/ServletContextHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.8.v20160314/jetty-servlet-9.3.8.v20160314.jar(org/eclipse/jetty/servlet/ServletHolder.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-xml/9.3.8.v20160314/jetty-xml-9.3.8.v20160314.jar(org/eclipse/jetty/xml/XmlConfiguration.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/slf4j/jul-to-slf4j/1.7.10/jul-to-slf4j-1.7.10.jar(org/slf4j/bridge/SLF4JBridgeHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/DispatcherType.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/Filter.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/FilterChain.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/FilterConfig.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/ServletException.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/ServletRequest.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/ServletResponse.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/annotation/WebFilter.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/http/HttpServletRequest.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/http/HttpServletResponse.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/classification/t
[jira] [Commented] (HIVE-19041) Thrift deserialization of Partition objects should intern fields
[
https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471832#comment-16471832
]
Hive QA commented on HIVE-19041:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922755/HIVE-19041.04.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10825/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10825/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10825/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-05-11 12:26:27.438
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-10825/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-05-11 12:26:27.440
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
fb79870..8028ce8 master -> origin/master
+ git reset --hard HEAD
HEAD is now at fb79870 HIVE-13745: UDF
current_date?current_timestamp?unix_timestamp NPE (Biao Wu, reviewed by Yongzhi
Chen)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 8028ce8 HIVE-19468: Add Apache license to TestTxnConcatenate
(Igor Kryvenko via Zoltan Haindrich)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-05-11 12:26:28.399
+ rm -rf ../yetus_PreCommit-HIVE-Build-10825
+ mkdir ../yetus_PreCommit-HIVE-Build-10825
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-10825
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-10825/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/standalone-metastore/pom.xml: does not exist in index
error:
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsDesc.java:
does not exist in index
error:
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsObj.java:
does not exist in index
error:
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java:
does not exist in index
error:
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java:
does not exist in index
error:
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java:
does not exist in index
error:
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java:
does not exist in index
error: a/standalone-metastore/src/main/resources/thrift-replacements.txt: does
not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc2144661588944428173.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc2144661588944428173.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/w
[jira] [Commented] (HIVE-19382) Acquire locks before generating valid transaction list for some operations
[
https://issues.apache.org/jira/browse/HIVE-19382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471865#comment-16471865
]
Hive QA commented on HIVE-19382:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 17 new + 170 unchanged - 6
fixed = 187 total (was 176) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
8s{color} | {color:red} ql generated 1 new + 2321 unchanged - 0 fixed = 2322
total (was 2321) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Redundant nullcheck of txnWriteIdList, which is known to be non-null in
org.apache.hadoop.hive.ql.Driver.isValidTxnListState() Redundant null check at
Driver.java:is known to be non-null in
org.apache.hadoop.hive.ql.Driver.isValidTxnListState() Redundant null check at
Driver.java:[line 912] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10826/dev-support/hive-personality.sh
|
| git revision | master / 8028ce8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10826/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10826/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10826/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Acquire locks before generating valid transaction list for some operations
> --
>
> Key: HIVE-19382
> URL: https://issues.apache.org/jira/browse/HIVE-19382
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19382.01.patch, HIVE-19382.02.patch,
> HIVE-19382.patch
>
>
> To ensure correctness, in particular for operations that require exclusive
> ({{INSERT OVERWRITE}}) and semishared ({{UPDATE}}/{{DELETE}}) locks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19501) Fix HyperLogLog to be threadsafe
[ https://issues.apache.org/jira/browse/HIVE-19501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471867#comment-16471867 ] Zoltan Haindrich commented on HIVE-19501: - also note that most probably the addShort / etc methods are unused > Fix HyperLogLog to be threadsafe > > > Key: HIVE-19501 > URL: https://issues.apache.org/jira/browse/HIVE-19501 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > > not sure if this is an issue in reality or not; but there are 3 static fields > in HyperLogLog which are rewritten during working; if there are multiple > threads are calculating HLL in the same JVM, there is a theoretical chance > that they might overwrite eachothers value... > static fields: > https://github.com/apache/hive/blob/8028ce8a4cf5a03e2998c33e032a511fae770b47/standalone-metastore/src/main/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLog.java#L65 > usage: > https://github.com/apache/hive/blob/8028ce8a4cf5a03e2998c33e032a511fae770b47/standalone-metastore/src/main/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLog.java#L216 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19460) Improve stats estimations for NOT IN operator
[ https://issues.apache.org/jira/browse/HIVE-19460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19460: Attachment: HIVE-19460.01wip03.patch > Improve stats estimations for NOT IN operator > - > > Key: HIVE-19460 > URL: https://issues.apache.org/jira/browse/HIVE-19460 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19460.01wip01.patch, HIVE-19460.01wip02.patch, > HIVE-19460.01wip03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19041) Thrift deserialization of Partition objects should intern fields
[ https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471921#comment-16471921 ] Sahil Takiar commented on HIVE-19041: - +1 latest patch LGTM, although looks like it needs to be rebased > Thrift deserialization of Partition objects should intern fields > > > Key: HIVE-19041 > URL: https://issues.apache.org/jira/browse/HIVE-19041 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 2.3.2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-19041.01.patch, HIVE-19041.02.patch, > HIVE-19041.03.patch, HIVE-19041.04.patch > > > When a client is creating large number of partitions, the thrift objects are > deserialized into Partition objects. The read method of these objects does > not intern the inputformat, location, outputformat which cause large number > of duplicate Strings in the HMS memory. We should intern these objects while > deserialization to reduce memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471926#comment-16471926 ] Sahil Takiar commented on HIVE-18117: - [~asherman] left some comments on the RB yesterday. Could you link the RB to this JIRA? > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch > > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19382) Acquire locks before generating valid transaction list for some operations
[
https://issues.apache.org/jira/browse/HIVE-19382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471995#comment-16471995
]
Hive QA commented on HIVE-19382:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922763/HIVE-19382.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 182 failed/errored test(s), 13399 tests
executed
*Failed tests:*
{noformat}
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=38)
[parquet_vectorization_11.q,vectorization_limit.q,autoColumnStats_9.q,partialdhj.q,drop_table_purge.q,partition_varchar2.q,avro_deserialize_map_null.q,rename_table_update_column_stats.q,bucketcontext_8.q,describe_database.q,input_testxpath2.q,stats_noscan_2.q,udf_mask_last_n.q,create_view_partitioned.q,input17.q,date_comparison.q,ppd_gby_join.q,udaf_binarysetfunctions.q,table_nonprintable.q,tez_union.q,literal_double.q,ppd_join.q,materialized_view_rewrite_ssb.q,dynpart_merge.q,udf8.q,udf_substr.q,comments.q,inputddl4.q,correlationoptimizer12.q,timestamp_comparison2.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=41)
[udf_radians.q,testdataset.q,orc_merge11.q,udf_float.q,cbo_semijoin.q,bucketmapjoin13.q,constant_prop_1.q,udf_split.q,macro_duplicate.q,load_dyn_part9.q,smb_mapjoin_8.q,kryo.q,partial_column_stats.q,auto_join16.q,tez_joins_explain.q,udf_replace.q,create_or_replace_view.q,alter_partition_clusterby_sortby.q,explaindenpendencydiffengs.q,transform2.q,vector_aggregate_9.q,exim_15_external_part.q,delete_orig_table.q,groupby_position.q,llap_acid_fast.q,acid_subquery.q,nullformatCTAS.q,join21.q,cbo_rp_groupby3_noskew_multi_distinct.q,delete_where_partitioned.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=57)
[groupby_rollup_empty.q,union29.q,quotedid_alter.q,udf_equal.q,infer_bucket_sort_grouping_operators.q,tez_union_dynamic_partition_2.q,exim_12_external_location.q,cbo_rp_join.q,quotedid_tblproperty.q,cte_2.q,autoColumnStats_3.q,parquet_array_of_unannotated_groups.q,fileformat_mix.q,alterColumnStats.q,describe_formatted_view_partitioned_json.q,vector_partition_diff_num_cols.q,dynamic_partition_insert.q,windowing_multipartitioning.q,schema_evol_text_vec_part.q,add_jar_pfile.q,windowing_expressions.q,auto_join29.q,interval_2.q,encryption_drop_table.q,notable_alias1.q,timestamptz_3.q,multi_insert_move_tasks_share_dependencies.q,rename_table_location.q,show_create_table_serde.q,list_bucket_dml_7.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=79)
[orc_ends_with_nulls.q,groupby_sort_7.q,type_conversions_1.q,parquet_vectorization_3.q,parquet_vectorization_div0.q,columnstats_infinity.q,floor_time.q,schema_evol_orc_vec_part_all_primitive.q,partition_wise_fileformat14.q,decimal_4.q,infer_bucket_sort_bucketed_table.q,cbo_union.q,reduce_deduplicate_exclude_join.q,schema_evol_orc_vec_part.q,ppd_transform.q,schema_evol_text_vec_part_all_primitive_llap_io.q,udf_pmod.q,vector_data_types.q,vector_ptf_1.q,correlationoptimizer10.q,cteViews.q,describe_formatted_view_partitioned.q,tez_union_with_udf.q,vectorization_3.q,disable_merge_for_bucketing.q,vectorized_date_funcs.q,create_view_defaultformats.q,vector_varchar_simple.q,groupby_multialias.q,named_column_join.q]
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestDbTxnManager2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=300)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHiveDecimalParse - did not produce a TEST-*.xml file (likely timed out)
(batchId=280)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,mm_update.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wro
[jira] [Assigned] (HIVE-19503) Create a test that checks that every table is empty after a dropTable happened with directSql
[ https://issues.apache.org/jira/browse/HIVE-19503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-19503: - > Create a test that checks that every table is empty after a dropTable > happened with directSql > - > > Key: HIVE-19503 > URL: https://issues.apache.org/jira/browse/HIVE-19503 > Project: Hive > Issue Type: Improvement > Components: Test >Affects Versions: 3.1.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > > As a followup jira, it might be good to check that every RDBMS table is empty > after a dropTable happened with DirectSQL -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19483) Metastore cleaner tasks that run periodically are created more than once
[
https://issues.apache.org/jira/browse/HIVE-19483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472014#comment-16472014
]
Hive QA commented on HIVE-19483:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
1s{color} | {color:blue} standalone-metastore in master has 215 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} standalone-metastore: The patch generated 2 new + 362
unchanged - 0 fixed = 364 total (was 362) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10827/dev-support/hive-personality.sh
|
| git revision | master / 8028ce8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10827/yetus/diff-checkstyle-standalone-metastore.txt
|
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10827/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Metastore cleaner tasks that run periodically are created more than once
>
>
> Key: HIVE-19483
> URL: https://issues.apache.org/jira/browse/HIVE-19483
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19483.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19435) Incremental replication cause data loss if a table is dropped followed by create and insert-into with different partition type.
[ https://issues.apache.org/jira/browse/HIVE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472036#comment-16472036 ] Sankar Hariappan commented on HIVE-19435: - [~thejas], Could you please review and +1 the patch? > Incremental replication cause data loss if a table is dropped followed by > create and insert-into with different partition type. > --- > > Key: HIVE-19435 > URL: https://issues.apache.org/jira/browse/HIVE-19435 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-19435.01.patch, HIVE-19435.02.patch, > HIVE-19435.03.patch > > > If the incremental dump have drop of partitioned table followed by > create/insert on non-partitioned table with same name, doesn't replicate the > data. Explained below. > Let's say we have a partitioned table T1 which was already replicated to > target. > DROP_TABLE(T1)->CREATE_TABLE(T1) (Non-partitioned) -> INSERT(T1)(10) > After REPL LOAD, T1 doesn't have any data. > Same is valid for non-partitioned to partitioned and partition spec mismatch > case as well. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19486) Discrepancy between the config and the code in Hikari connectionPoolingType
[ https://issues.apache.org/jira/browse/HIVE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-19486: --- Description: MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" while the code checks for "Hikari". (was: MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" not "Hikari".) > Discrepancy between the config and the code in Hikari connectionPoolingType > --- > > Key: HIVE-19486 > URL: https://issues.apache.org/jira/browse/HIVE-19486 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" while the > code checks for "Hikari". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19483) Metastore cleaner tasks that run periodically are created more than once
[
https://issues.apache.org/jira/browse/HIVE-19483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472127#comment-16472127
]
Hive QA commented on HIVE-19483:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922773/HIVE-19483.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 31 failed/errored test(s), 13576 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,mm_update.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,udf_sort_array_by_wrong2.q,local_
[jira] [Work started] (HIVE-19486) Discrepancy between the config and the code in Hikari connectionPoolingType
[ https://issues.apache.org/jira/browse/HIVE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-19486 started by Antal Sinkovits. -- > Discrepancy between the config and the code in Hikari connectionPoolingType > --- > > Key: HIVE-19486 > URL: https://issues.apache.org/jira/browse/HIVE-19486 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-19486.1.patch > > > MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" while the > code checks for "Hikari". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19486) Discrepancy between the config and the code in Hikari connectionPoolingType
[ https://issues.apache.org/jira/browse/HIVE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-19486: --- Status: Patch Available (was: In Progress) > Discrepancy between the config and the code in Hikari connectionPoolingType > --- > > Key: HIVE-19486 > URL: https://issues.apache.org/jira/browse/HIVE-19486 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-19486.1.patch > > > MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" while the > code checks for "Hikari". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19486) Discrepancy between the config and the code in Hikari connectionPoolingType
[ https://issues.apache.org/jira/browse/HIVE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-19486: --- Attachment: HIVE-19486.1.patch > Discrepancy between the config and the code in Hikari connectionPoolingType > --- > > Key: HIVE-19486 > URL: https://issues.apache.org/jira/browse/HIVE-19486 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-19486.1.patch > > > MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" while the > code checks for "Hikari". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19435) Incremental replication cause data loss if a table is dropped followed by create and insert-into with different partition type.
[ https://issues.apache.org/jira/browse/HIVE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472165#comment-16472165 ] Thejas M Nair commented on HIVE-19435: -- +1 > Incremental replication cause data loss if a table is dropped followed by > create and insert-into with different partition type. > --- > > Key: HIVE-19435 > URL: https://issues.apache.org/jira/browse/HIVE-19435 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-19435.01.patch, HIVE-19435.02.patch, > HIVE-19435.03.patch > > > If the incremental dump have drop of partitioned table followed by > create/insert on non-partitioned table with same name, doesn't replicate the > data. Explained below. > Let's say we have a partitioned table T1 which was already replicated to > target. > DROP_TABLE(T1)->CREATE_TABLE(T1) (Non-partitioned) -> INSERT(T1)(10) > After REPL LOAD, T1 doesn't have any data. > Same is valid for non-partitioned to partitioned and partition spec mismatch > case as well. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19455) Create JDBC External Table NullPointerException
[
https://issues.apache.org/jira/browse/HIVE-19455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472167#comment-16472167
]
Hive QA commented on HIVE-19455:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} jdbc-handler in master has 8 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10829/dev-support/hive-personality.sh
|
| git revision | master / 8028ce8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: jdbc-handler U: jdbc-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10829/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create JDBC External Table NullPointerException
> ---
>
> Key: HIVE-19455
> URL: https://issues.apache.org/jira/browse/HIVE-19455
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.0.0, 2.3.3
>Reporter: gonglinglei
>Priority: Major
> Attachments: HIVE-19455.1.patch, HIVE-19455.2.patch
>
>
> {{JdbcSerDe.initialize}} use
> {{tbl.containsKey(JdbcStorageConfig.DATABASE_TYPE.getPropertyName())}} to
> decide whether properties is empty and whether to initialize serde. But when
> creating a external table
> without {{hive.sql.database.type}} provided, it will throw a
> NullPointerException.
> {quote}
> 2018-05-08T11:21:03,745 ERROR [88c8bc6c-cd5b-4b74-b6d6-242e3cc12165 main]
> metadata.Table: Unable to get field from serde:
> org.apache.hive.storage.jdbc.JdbcSerDe
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getFieldsFromDeserializer(MetaStoreUtils.java:1426)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:641)
> ~[hive-exec-2.3.3.jar:2.3.3]
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19317) Handle schema evolution from int like types to decimal
[ https://issues.apache.org/jira/browse/HIVE-19317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472185#comment-16472185 ] Vihang Karajgaonkar commented on HIVE-19317: Thanks [~janulatha] for the patch. Overall looks good to me. I left some suggestions on the review board. > Handle schema evolution from int like types to decimal > -- > > Key: HIVE-19317 > URL: https://issues.apache.org/jira/browse/HIVE-19317 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-19317.1.patch, HIVE-19317.2.patch, > HIVE-19317.3.patch, HIVE-19317.4.patch > > > If int like type is changed to decimal on parquet data, select results in > errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19477) Hiveserver2 in http mode not emitting metric default.General.open_connections
[ https://issues.apache.org/jira/browse/HIVE-19477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-19477: --- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master, branch-3. Thanks [~vgumashta] > Hiveserver2 in http mode not emitting metric default.General.open_connections > - > > Key: HIVE-19477 > URL: https://issues.apache.org/jira/browse/HIVE-19477 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Dinesh Chitlangia >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-19477.01.patch, HIVE-19477.patch > > > Instances in binary mode are emitting the metric > _default.General.open_connections_ but the instances operating in http mode > are not emitting this metric. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19483) Metastore cleaner tasks that run periodically are created more than once
[ https://issues.apache.org/jira/browse/HIVE-19483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-19483: --- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master, branch-3. Thanks [~alangates] > Metastore cleaner tasks that run periodically are created more than once > > > Key: HIVE-19483 > URL: https://issues.apache.org/jira/browse/HIVE-19483 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19483.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16949) Leak of threads from Get-Input-Paths and Get-Input-Summary thread pool
[
https://issues.apache.org/jira/browse/HIVE-16949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16949:
Fix Version/s: 3.0.0
> Leak of threads from Get-Input-Paths and Get-Input-Summary thread pool
> --
>
> Key: HIVE-16949
> URL: https://issues.apache.org/jira/browse/HIVE-16949
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Birger Brunswiek
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-16949.1.patch
>
>
> The commit
> [20210de|https://github.com/apache/hive/commit/20210dec94148c9b529132b1545df3dd7be083c3]
> which was part of HIVE-15546 [introduced a thread
> pool|https://github.com/apache/hive/blob/824b9c80b443dc4e2b9ad35214a23ac756e75234/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L3109]
> which is not shutdown upon completion of its threads. This leads to a leak
> of threads for each query which uses more than 1 partition. They are not
> removed automatically. When queries spanning multiple partitions are made the
> number of threads increases and is never reduced. On my machine hiveserver2
> starts to get slower and slower once 10k threads are reached.
> Thread pools only shutdown automatically in special circumstances (see
> [documentation section
> _Finalization_|https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadPoolExecutor.html]).
> This is not currently the case for the Get-Input-Paths thread pool. I would
> add a _pool.shutdown()_ in a finally block just before returning the result
> to make sure the threads are really shutdown.
> My current workaround is to set {{hive.exec.input.listing.max.threads = 1}}.
> This prevents the the thread pool from being spawned
> [\[1\]|https://github.com/apache/hive/blob/824b9c80b443dc4e2b9ad35214a23ac756e75234/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2118]
>
> [\[2\]|https://github.com/apache/hive/blob/824b9c80b443dc4e2b9ad35214a23ac756e75234/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L3107].
> The same issue probably also applies to the [Get-Input-Summary thread
> pool|https://github.com/apache/hive/blob/824b9c80b443dc4e2b9ad35214a23ac756e75234/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2193].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19435) Incremental replication cause data loss if a table is dropped followed by create and insert-into with different partition type.
[ https://issues.apache.org/jira/browse/HIVE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19435: Status: Open (was: Patch Available) > Incremental replication cause data loss if a table is dropped followed by > create and insert-into with different partition type. > --- > > Key: HIVE-19435 > URL: https://issues.apache.org/jira/browse/HIVE-19435 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-19435.01-branch-3.patch, HIVE-19435.01.patch, > HIVE-19435.02.patch, HIVE-19435.03.patch > > > If the incremental dump have drop of partitioned table followed by > create/insert on non-partitioned table with same name, doesn't replicate the > data. Explained below. > Let's say we have a partitioned table T1 which was already replicated to > target. > DROP_TABLE(T1)->CREATE_TABLE(T1) (Non-partitioned) -> INSERT(T1)(10) > After REPL LOAD, T1 doesn't have any data. > Same is valid for non-partitioned to partitioned and partition spec mismatch > case as well. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19481) sample10.q returns possibly wrong results for insert-only transactional table
[ https://issues.apache.org/jira/browse/HIVE-19481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16469683#comment-16469683 ] Steve Yeom edited comment on HIVE-19481 at 5/11/18 5:01 PM: The semantics of the test looks like the results for insert-only transactional table is rather valid to return couple of rows for the above two queries. I.e., possibly non-transactional regular table return results of no row may be incorrect. was (Author: steveyeom2017): The query on insert-only transactional table returns no rows (or correct) when vectorization is off. > sample10.q returns possibly wrong results for insert-only transactional table > - > > Key: HIVE-19481 > URL: https://issues.apache.org/jira/browse/HIVE-19481 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 3.1.0 > > > Ran "mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=sample10.q " after > changing the table to be > insert-only transactional. > The following queries returns couple of rows whereis no row results returns > for non-ACID table. > query: select ds, count(1) from srcpartbucket tablesample (bucket 2 out of 4 > on key) where ds is not null group by ds ORDER BY ds ASC > 2008-04-08 14 > 2008-04-09 14 > .. > query: select ds, count(1) from srcpartbucket tablesample (bucket 1 out of 2 > on key) where ds is not null group by ds ORDER BY ds ASC > 2008-04-08 4 > 2008-04-09 4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19435) Incremental replication cause data loss if a table is dropped followed by create and insert-into with different partition type.
[ https://issues.apache.org/jira/browse/HIVE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19435: Attachment: HIVE-19435.01-branch-3.patch > Incremental replication cause data loss if a table is dropped followed by > create and insert-into with different partition type. > --- > > Key: HIVE-19435 > URL: https://issues.apache.org/jira/browse/HIVE-19435 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-19435.01-branch-3.patch, HIVE-19435.01.patch, > HIVE-19435.02.patch, HIVE-19435.03.patch > > > If the incremental dump have drop of partitioned table followed by > create/insert on non-partitioned table with same name, doesn't replicate the > data. Explained below. > Let's say we have a partitioned table T1 which was already replicated to > target. > DROP_TABLE(T1)->CREATE_TABLE(T1) (Non-partitioned) -> INSERT(T1)(10) > After REPL LOAD, T1 doesn't have any data. > Same is valid for non-partitioned to partitioned and partition spec mismatch > case as well. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19435) Incremental replication cause data loss if a table is dropped followed by create and insert-into with different partition type.
[ https://issues.apache.org/jira/browse/HIVE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19435: Fix Version/s: 3.0.0 > Incremental replication cause data loss if a table is dropped followed by > create and insert-into with different partition type. > --- > > Key: HIVE-19435 > URL: https://issues.apache.org/jira/browse/HIVE-19435 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19435.01-branch-3.patch, HIVE-19435.01.patch, > HIVE-19435.02.patch, HIVE-19435.03.patch > > > If the incremental dump have drop of partitioned table followed by > create/insert on non-partitioned table with same name, doesn't replicate the > data. Explained below. > Let's say we have a partitioned table T1 which was already replicated to > target. > DROP_TABLE(T1)->CREATE_TABLE(T1) (Non-partitioned) -> INSERT(T1)(10) > After REPL LOAD, T1 doesn't have any data. > Same is valid for non-partitioned to partitioned and partition spec mismatch > case as well. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19435) Incremental replication cause data loss if a table is dropped followed by create and insert-into with different partition type.
[ https://issues.apache.org/jira/browse/HIVE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19435: Status: Patch Available (was: Open) Thanks for the review [~thejas] and [~maheshk114]! Attached 01.patch for branch-3. > Incremental replication cause data loss if a table is dropped followed by > create and insert-into with different partition type. > --- > > Key: HIVE-19435 > URL: https://issues.apache.org/jira/browse/HIVE-19435 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-19435.01-branch-3.patch, HIVE-19435.01.patch, > HIVE-19435.02.patch, HIVE-19435.03.patch > > > If the incremental dump have drop of partitioned table followed by > create/insert on non-partitioned table with same name, doesn't replicate the > data. Explained below. > Let's say we have a partitioned table T1 which was already replicated to > target. > DROP_TABLE(T1)->CREATE_TABLE(T1) (Non-partitioned) -> INSERT(T1)(10) > After REPL LOAD, T1 doesn't have any data. > Same is valid for non-partitioned to partitioned and partition spec mismatch > case as well. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19481) sample10.q returns possibly wrong results for insert-only transactional table
[ https://issues.apache.org/jira/browse/HIVE-19481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472284#comment-16472284 ] Steve Yeom commented on HIVE-19481: --- It looks like the results of the test returns two rows for the two queries on non-transactional table before patch of HIVE-18910 > sample10.q returns possibly wrong results for insert-only transactional table > - > > Key: HIVE-19481 > URL: https://issues.apache.org/jira/browse/HIVE-19481 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 3.1.0 > > > Ran "mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=sample10.q " after > changing the table to be > insert-only transactional. > The following queries returns couple of rows whereis no row results returns > for non-ACID table. > query: select ds, count(1) from srcpartbucket tablesample (bucket 2 out of 4 > on key) where ds is not null group by ds ORDER BY ds ASC > 2008-04-08 14 > 2008-04-09 14 > .. > query: select ds, count(1) from srcpartbucket tablesample (bucket 1 out of 2 > on key) where ds is not null group by ds ORDER BY ds ASC > 2008-04-08 4 > 2008-04-09 4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19500) Prevent multiple selectivity estimations for the same variable in conjuctions
[
https://issues.apache.org/jira/browse/HIVE-19500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19500:
Attachment: HIVE-19500.01.patch
> Prevent multiple selectivity estimations for the same variable in conjuctions
> -
>
> Key: HIVE-19500
> URL: https://issues.apache.org/jira/browse/HIVE-19500
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19500.01.patch
>
>
> see HIVE-19097 for problem description
> for filters like: {{(d_year in (2001,2002) and d_year = 2001)}} the current
> estimation is around {{(1/NDV)**2}} (iff column stats are available)
> this patch targets on branch-2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19500) Prevent multiple selectivity estimations for the same variable in conjuctions
[
https://issues.apache.org/jira/browse/HIVE-19500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19500:
Affects Version/s: 3.1.0
3.0.0
Target Version/s: 3.0.0, 3.1.0 (was: 2.3.2)
Description:
see HIVE-19097 for problem description
for filters like: {{(d_year in (2001,2002) and d_year = 2001)}} the current
estimation is around {{(1/NDV)**2}} (iff column stats are available)
actually the source of the problem was a small typo in HIVE-17465
was:
see HIVE-19097 for problem description
for filters like: {{(d_year in (2001,2002) and d_year = 2001)}} the current
estimation is around {{(1/NDV)**2}} (iff column stats are available)
this patch targets on branch-2
> Prevent multiple selectivity estimations for the same variable in conjuctions
> -
>
> Key: HIVE-19500
> URL: https://issues.apache.org/jira/browse/HIVE-19500
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19500.01.patch
>
>
> see HIVE-19097 for problem description
> for filters like: {{(d_year in (2001,2002) and d_year = 2001)}} the current
> estimation is around {{(1/NDV)**2}} (iff column stats are available)
> actually the source of the problem was a small typo in HIVE-17465
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19500) Prevent multiple selectivity estimations for the same variable in conjuctions
[
https://issues.apache.org/jira/browse/HIVE-19500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19500:
Status: Patch Available (was: Open)
> Prevent multiple selectivity estimations for the same variable in conjuctions
> -
>
> Key: HIVE-19500
> URL: https://issues.apache.org/jira/browse/HIVE-19500
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19500.01.patch
>
>
> see HIVE-19097 for problem description
> for filters like: {{(d_year in (2001,2002) and d_year = 2001)}} the current
> estimation is around {{(1/NDV)**2}} (iff column stats are available)
> actually the source of the problem was a small typo in HIVE-17465
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19455) Create JDBC External Table NullPointerException
[
https://issues.apache.org/jira/browse/HIVE-19455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472290#comment-16472290
]
Hive QA commented on HIVE-19455:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922779/HIVE-19455.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 30 failed/errored test(s), 13576 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,mm_update.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,udf_sort_array_by_wrong2.q,loca
[jira] [Commented] (HIVE-14388) Add number of rows inserted message after insert command in Beeline
[ https://issues.apache.org/jira/browse/HIVE-14388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472331#comment-16472331 ] Bharathkrishna Guruvayoor Murali commented on HIVE-14388: - Thanks Sahil for the review. Updating the latest version of patch from review board. > Add number of rows inserted message after insert command in Beeline > --- > > Key: HIVE-14388 > URL: https://issues.apache.org/jira/browse/HIVE-14388 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vihang Karajgaonkar >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Minor > Attachments: HIVE-14388-WIP.patch, HIVE-14388.02.patch, > HIVE-14388.03.patch, HIVE-14388.05.patch, HIVE-14388.06.patch, > HIVE-14388.07.patch, HIVE-14388.08.patch, HIVE-14388.09.patch, > HIVE-14388.10.patch > > > Currently, when you run insert command on beeline, it returns a message > saying "No rows affected .." > A better and more intuitive msg would be "xxx rows inserted (26.068 seconds)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14388) Add number of rows inserted message after insert command in Beeline
[ https://issues.apache.org/jira/browse/HIVE-14388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-14388: Attachment: HIVE-14388.10.patch > Add number of rows inserted message after insert command in Beeline > --- > > Key: HIVE-14388 > URL: https://issues.apache.org/jira/browse/HIVE-14388 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vihang Karajgaonkar >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Minor > Attachments: HIVE-14388-WIP.patch, HIVE-14388.02.patch, > HIVE-14388.03.patch, HIVE-14388.05.patch, HIVE-14388.06.patch, > HIVE-14388.07.patch, HIVE-14388.08.patch, HIVE-14388.09.patch, > HIVE-14388.10.patch > > > Currently, when you run insert command on beeline, it returns a message > saying "No rows affected .." > A better and more intuitive msg would be "xxx rows inserted (26.068 seconds)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19384) Vectorization: IfExprTimestamp* do not handle NULLs correctly
[
https://issues.apache.org/jira/browse/HIVE-19384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472346#comment-16472346
]
Hive QA commented on HIVE-19384:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
31s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} storage-api: The patch generated 2 new + 12 unchanged
- 2 fixed = 14 total (was 14) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 75 new + 606 unchanged - 16
fixed = 681 total (was 622) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} vector-code-gen: The patch generated 3 new + 316
unchanged - 0 fixed = 319 total (was 316) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
28s{color} | {color:red} ql generated 8 new + 2320 unchanged - 1 fixed = 2328
total (was 2321) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Class
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.IfExprIntervalDayTimeColumnScalar
defines non-transient non-serializable instance field arg3Scalar In
IfExprIntervalDayTimeColumnScalar.java:instance field arg3Scalar In
IfExprIntervalDayTimeColumnScalar.java |
| | Class
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.IfExprIntervalDayTimeScalarColumn
defines non-transient non-serializable instance field arg2Scalar In
IfExprIntervalDayTimeScalarColumn.java:instance field arg2Scalar In
IfExprIntervalDayTimeScalarColumn.java |
| | Class
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.IfExprIntervalDayTimeScalarScalar
defines non-transient non-serializable instance field arg2Scalar In
IfExprIntervalDayTimeScalarScalar.java:instance field arg2Scalar In
IfExprIntervalDayTimeScalarScalar.java |
| | Class
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.IfExprIntervalDayTimeScalarScalar
defines non-transient non-serializable instance field arg3Scalar In
IfExprIntervalDayTimeScalarScalar.java:instance field arg3Scalar In
IfExprIntervalDayTimeScalarScalar.java |
| | new
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.IfExprTimestampColumnScalar(int,
int, Timestamp, int) may expose internal representation by storing an
externally mutable object into IfExprTimestampColumnScalar.arg3Scalar At
If
[jira] [Work started] (HIVE-18394) Materialized view: "Create Materialized View" should default to rewritable ones
[
https://issues.apache.org/jira/browse/HIVE-18394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-18394 started by Jesus Camacho Rodriguez.
--
> Materialized view: "Create Materialized View" should default to rewritable
> ones
> ---
>
> Key: HIVE-18394
> URL: https://issues.apache.org/jira/browse/HIVE-18394
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Attachments: HIVE-18394.patch
>
>
> This is a usability ticket, since it is possible to end up creating
> materialized views and realize that they need an additional flag to be picked
> up by the optimizer to do rewrites to.
> {code:sql}
> create materialized view ca as select * from customer, customer_address where
> c_current_addr_sk = ca_address_sk;
> set hive.materializedview.rewriting=true;
> select count(1) from customer, customer_address where c_current_addr_sk =
> ca_address_sk; -- does not use materialized view
> {code}
> Needs another step
> {code:sql}
> alter materialized view ca enable rewrite;
> {code}
> And then, it kicks in
> {code:sql}
> select count(1) from customer, customer_address where c_current_addr_sk =
> ca_address_sk;
> OK
> 1200
> Time taken: 0.494 seconds, Fetched: 1 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18394) Materialized view: "Create Materialized View" should default to rewritable ones
[
https://issues.apache.org/jira/browse/HIVE-18394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472347#comment-16472347
]
Jesus Camacho Rodriguez commented on HIVE-18394:
[~gopalv], could you review the patch? Thanks
> Materialized view: "Create Materialized View" should default to rewritable
> ones
> ---
>
> Key: HIVE-18394
> URL: https://issues.apache.org/jira/browse/HIVE-18394
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Attachments: HIVE-18394.patch
>
>
> This is a usability ticket, since it is possible to end up creating
> materialized views and realize that they need an additional flag to be picked
> up by the optimizer to do rewrites to.
> {code:sql}
> create materialized view ca as select * from customer, customer_address where
> c_current_addr_sk = ca_address_sk;
> set hive.materializedview.rewriting=true;
> select count(1) from customer, customer_address where c_current_addr_sk =
> ca_address_sk; -- does not use materialized view
> {code}
> Needs another step
> {code:sql}
> alter materialized view ca enable rewrite;
> {code}
> And then, it kicks in
> {code:sql}
> select count(1) from customer, customer_address where c_current_addr_sk =
> ca_address_sk;
> OK
> 1200
> Time taken: 0.494 seconds, Fetched: 1 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19326) union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats (incorrect query results possible)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472348#comment-16472348
]
Vineet Garg commented on HIVE-19326:
So is this just test issue?
> union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats
> (incorrect query results possible)
>
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Blocker
> Fix For: 3.0.0
>
>
> Found when investigating the results change after converting tables to MM,
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 5
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 15
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles2
> numRows 20
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate,
> but BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect
> results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18394) Materialized view: "Create Materialized View" should default to rewritable ones
[
https://issues.apache.org/jira/browse/HIVE-18394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-18394:
---
Attachment: HIVE-18394.patch
> Materialized view: "Create Materialized View" should default to rewritable
> ones
> ---
>
> Key: HIVE-18394
> URL: https://issues.apache.org/jira/browse/HIVE-18394
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Attachments: HIVE-18394.patch
>
>
> This is a usability ticket, since it is possible to end up creating
> materialized views and realize that they need an additional flag to be picked
> up by the optimizer to do rewrites to.
> {code:sql}
> create materialized view ca as select * from customer, customer_address where
> c_current_addr_sk = ca_address_sk;
> set hive.materializedview.rewriting=true;
> select count(1) from customer, customer_address where c_current_addr_sk =
> ca_address_sk; -- does not use materialized view
> {code}
> Needs another step
> {code:sql}
> alter materialized view ca enable rewrite;
> {code}
> And then, it kicks in
> {code:sql}
> select count(1) from customer, customer_address where c_current_addr_sk =
> ca_address_sk;
> OK
> 1200
> Time taken: 0.494 seconds, Fetched: 1 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18394) Materialized view: "Create Materialized View" should default to rewritable ones
[
https://issues.apache.org/jira/browse/HIVE-18394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-18394:
---
Status: Patch Available (was: In Progress)
> Materialized view: "Create Materialized View" should default to rewritable
> ones
> ---
>
> Key: HIVE-18394
> URL: https://issues.apache.org/jira/browse/HIVE-18394
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Attachments: HIVE-18394.patch
>
>
> This is a usability ticket, since it is possible to end up creating
> materialized views and realize that they need an additional flag to be picked
> up by the optimizer to do rewrites to.
> {code:sql}
> create materialized view ca as select * from customer, customer_address where
> c_current_addr_sk = ca_address_sk;
> set hive.materializedview.rewriting=true;
> select count(1) from customer, customer_address where c_current_addr_sk =
> ca_address_sk; -- does not use materialized view
> {code}
> Needs another step
> {code:sql}
> alter materialized view ca enable rewrite;
> {code}
> And then, it kicks in
> {code:sql}
> select count(1) from customer, customer_address where c_current_addr_sk =
> ca_address_sk;
> OK
> 1200
> Time taken: 0.494 seconds, Fetched: 1 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19500) Prevent multiple selectivity estimations for the same variable in conjuctions
[
https://issues.apache.org/jira/browse/HIVE-19500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472350#comment-16472350
]
Zoltan Haindrich commented on HIVE-19500:
-
[~vgarg] I'm not sure if this was a typo in HIVE-17465 ; or it was part of the
intended change
> Prevent multiple selectivity estimations for the same variable in conjuctions
> -
>
> Key: HIVE-19500
> URL: https://issues.apache.org/jira/browse/HIVE-19500
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19500.01.patch
>
>
> see HIVE-19097 for problem description
> for filters like: {{(d_year in (2001,2002) and d_year = 2001)}} the current
> estimation is around {{(1/NDV)**2}} (iff column stats are available)
> actually the source of the problem was a small typo in HIVE-17465
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19497) SessionHiveMetaStoreClient.getTable should respect default catalog
[ https://issues.apache.org/jira/browse/HIVE-19497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472352#comment-16472352 ] Alan Gates commented on HIVE-19497: --- +1, pending tests. [~vgarg] I need to put this in branch-3 as well > SessionHiveMetaStoreClient.getTable should respect default catalog > -- > > Key: HIVE-19497 > URL: https://issues.apache.org/jira/browse/HIVE-19497 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Priority: Major > Attachments: > 0001-HIVE-19497-SessionHiveMetaStoreClient.getTable-shoul.patch > > > Currently, SessionHiveMetaStoreClient always use `DEFAULT_CATALOG_NAME` > during looking up the underlying tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19326) union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats (incorrect query results possible)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472355#comment-16472355
]
Zoltan Haindrich commented on HIVE-19326:
-
no, this is a stats optimizer bug
> union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats
> (incorrect query results possible)
>
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Blocker
> Fix For: 3.0.0
>
>
> Found when investigating the results change after converting tables to MM,
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 5
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 15
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles2
> numRows 20
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate,
> but BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect
> results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19497) SessionHiveMetaStoreClient.getTable should respect default catalog
[ https://issues.apache.org/jira/browse/HIVE-19497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-19497: -- Priority: Blocker (was: Major) > SessionHiveMetaStoreClient.getTable should respect default catalog > -- > > Key: HIVE-19497 > URL: https://issues.apache.org/jira/browse/HIVE-19497 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Priority: Blocker > Attachments: > 0001-HIVE-19497-SessionHiveMetaStoreClient.getTable-shoul.patch > > > Currently, SessionHiveMetaStoreClient always use `DEFAULT_CATALOG_NAME` > during looking up the underlying tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19504) Change default value for hive.auto.convert.join.shuffle.max.size property
[
https://issues.apache.org/jira/browse/HIVE-19504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-19504:
--
> Change default value for hive.auto.convert.join.shuffle.max.size property
> -
>
> Key: HIVE-19504
> URL: https://issues.apache.org/jira/browse/HIVE-19504
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> The property default value is too low by mistake (10MB), it is missing three
> trailing zeros.
> {code}
> HIVECONVERTJOINMAXSHUFFLESIZE("hive.auto.convert.join.shuffle.max.size",
> 1000L,
>"If hive.auto.convert.join.noconditionaltask is off, this parameter
> does not take affect. \n" +
>"However, if it is on, and the predicted size of the larger input for
> a given join is greater \n" +
>"than this number, the join will not be converted to a dynamically
> partitioned hash join. \n" +
>"The value \"-1\" means no limit."),
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-19504) Change default value for hive.auto.convert.join.shuffle.max.size property
[
https://issues.apache.org/jira/browse/HIVE-19504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-19504 started by Jesus Camacho Rodriguez.
--
> Change default value for hive.auto.convert.join.shuffle.max.size property
> -
>
> Key: HIVE-19504
> URL: https://issues.apache.org/jira/browse/HIVE-19504
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19504.patch
>
>
> The property default value is too low by mistake (10MB), it is missing three
> trailing zeros.
> {code}
> HIVECONVERTJOINMAXSHUFFLESIZE("hive.auto.convert.join.shuffle.max.size",
> 1000L,
>"If hive.auto.convert.join.noconditionaltask is off, this parameter
> does not take affect. \n" +
>"However, if it is on, and the predicted size of the larger input for
> a given join is greater \n" +
>"than this number, the join will not be converted to a dynamically
> partitioned hash join. \n" +
>"The value \"-1\" means no limit."),
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19504) Change default value for hive.auto.convert.join.shuffle.max.size property
[
https://issues.apache.org/jira/browse/HIVE-19504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19504:
---
Status: Patch Available (was: In Progress)
> Change default value for hive.auto.convert.join.shuffle.max.size property
> -
>
> Key: HIVE-19504
> URL: https://issues.apache.org/jira/browse/HIVE-19504
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19504.patch
>
>
> The property default value is too low by mistake (10MB), it is missing three
> trailing zeros.
> {code}
> HIVECONVERTJOINMAXSHUFFLESIZE("hive.auto.convert.join.shuffle.max.size",
> 1000L,
>"If hive.auto.convert.join.noconditionaltask is off, this parameter
> does not take affect. \n" +
>"However, if it is on, and the predicted size of the larger input for
> a given join is greater \n" +
>"than this number, the join will not be converted to a dynamically
> partitioned hash join. \n" +
>"The value \"-1\" means no limit."),
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19504) Change default value for hive.auto.convert.join.shuffle.max.size property
[
https://issues.apache.org/jira/browse/HIVE-19504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472357#comment-16472357
]
Jesus Camacho Rodriguez commented on HIVE-19504:
[~vgarg], could you take a look? Thanks
> Change default value for hive.auto.convert.join.shuffle.max.size property
> -
>
> Key: HIVE-19504
> URL: https://issues.apache.org/jira/browse/HIVE-19504
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19504.patch
>
>
> The property default value is too low by mistake (10MB), it is missing three
> trailing zeros.
> {code}
> HIVECONVERTJOINMAXSHUFFLESIZE("hive.auto.convert.join.shuffle.max.size",
> 1000L,
>"If hive.auto.convert.join.noconditionaltask is off, this parameter
> does not take affect. \n" +
>"However, if it is on, and the predicted size of the larger input for
> a given join is greater \n" +
>"than this number, the join will not be converted to a dynamically
> partitioned hash join. \n" +
>"The value \"-1\" means no limit."),
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19504) Change default value for hive.auto.convert.join.shuffle.max.size property
[
https://issues.apache.org/jira/browse/HIVE-19504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19504:
---
Attachment: HIVE-19504.patch
> Change default value for hive.auto.convert.join.shuffle.max.size property
> -
>
> Key: HIVE-19504
> URL: https://issues.apache.org/jira/browse/HIVE-19504
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19504.patch
>
>
> The property default value is too low by mistake (10MB), it is missing three
> trailing zeros.
> {code}
> HIVECONVERTJOINMAXSHUFFLESIZE("hive.auto.convert.join.shuffle.max.size",
> 1000L,
>"If hive.auto.convert.join.noconditionaltask is off, this parameter
> does not take affect. \n" +
>"However, if it is on, and the predicted size of the larger input for
> a given join is greater \n" +
>"than this number, the join will not be converted to a dynamically
> partitioned hash join. \n" +
>"The value \"-1\" means no limit."),
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19453) Extend Load Data statement to take Input file format and Serde as parameters
[ https://issues.apache.org/jira/browse/HIVE-19453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472360#comment-16472360 ] Deepak Jaiswal commented on HIVE-19453: --- committed to branch-3. > Extend Load Data statement to take Input file format and Serde as parameters > > > Key: HIVE-19453 > URL: https://issues.apache.org/jira/browse/HIVE-19453 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-19453.01-branch-3.patch, HIVE-19453.1.patch > > > Extend the load data statement to take the inputformat of the source files > and the serde to interpret it as parameter. For eg, > > load data local inpath > '../../data/files/load_data_job/partitions/load_data_2_partitions.txt' INTO > TABLE srcbucket_mapjoin > INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' > SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19326) union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats (incorrect query results possible)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472363#comment-16472363
]
Sergey Shelukhin commented on HIVE-19326:
-
This is incorrect results bug in the default config (metadata-only queries
enabled)
> union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats
> (incorrect query results possible)
>
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Blocker
> Fix For: 3.0.0
>
>
> Found when investigating the results change after converting tables to MM,
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 5
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 15
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles2
> numRows 20
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate,
> but BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect
> results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18652) Print Spark metrics on console
[
https://issues.apache.org/jira/browse/HIVE-18652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18652:
Attachment: HIVE-18652.3.patch
> Print Spark metrics on console
> --
>
> Key: HIVE-18652
> URL: https://issues.apache.org/jira/browse/HIVE-18652
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18652.1.patch, HIVE-18652.2.patch,
> HIVE-18652.3.patch
>
>
> For Hive-on-MR, each MR job launched prints out some stats about the job:
> {code}
> INFO : 2018-02-07 17:51:11,218 Stage-1 map = 0%, reduce = 0%
> INFO : 2018-02-07 17:51:18,396 Stage-1 map = 100%, reduce = 0%, Cumulative
> CPU 1.87 sec
> INFO : 2018-02-07 17:51:25,742 Stage-1 map = 100%, reduce = 100%,
> Cumulative CPU 4.34 sec
> INFO : MapReduce Total cumulative CPU time: 4 seconds 340 msec
> INFO : Ended Job = job_1517865654989_0004
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 4.34 sec HDFS
> Read: 7353 HDFS Write: 151 SUCCESS
> INFO : Total MapReduce CPU Time Spent: 4 seconds 340 msec
> {code}
> We should do the same for Hive-on-Spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18652) Print Spark metrics on console
[
https://issues.apache.org/jira/browse/HIVE-18652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472368#comment-16472368
]
Sahil Takiar commented on HIVE-18652:
-
Thanks [~vihangk1]. Addressed your comments and updated the RB. Let me know if
you have any other comments, otherwise will wait for Hive QA results and then
merge this.
> Print Spark metrics on console
> --
>
> Key: HIVE-18652
> URL: https://issues.apache.org/jira/browse/HIVE-18652
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18652.1.patch, HIVE-18652.2.patch,
> HIVE-18652.3.patch
>
>
> For Hive-on-MR, each MR job launched prints out some stats about the job:
> {code}
> INFO : 2018-02-07 17:51:11,218 Stage-1 map = 0%, reduce = 0%
> INFO : 2018-02-07 17:51:18,396 Stage-1 map = 100%, reduce = 0%, Cumulative
> CPU 1.87 sec
> INFO : 2018-02-07 17:51:25,742 Stage-1 map = 100%, reduce = 100%,
> Cumulative CPU 4.34 sec
> INFO : MapReduce Total cumulative CPU time: 4 seconds 340 msec
> INFO : Ended Job = job_1517865654989_0004
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 4.34 sec HDFS
> Read: 7353 HDFS Write: 151 SUCCESS
> INFO : Total MapReduce CPU Time Spent: 4 seconds 340 msec
> {code}
> We should do the same for Hive-on-Spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19326) union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats (incorrect query results possible)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472369#comment-16472369
]
Vineet Garg commented on HIVE-19326:
Thanks for confirming. Is someone working on a patch for this? If not let us
push this out of 3.0 and fix this in bug fix release.
> union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats
> (incorrect query results possible)
>
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Blocker
> Fix For: 3.0.0
>
>
> Found when investigating the results change after converting tables to MM,
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 5
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 15
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles2
> numRows 20
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate,
> but BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect
> results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19504) Change default value for hive.auto.convert.join.shuffle.max.size property
[
https://issues.apache.org/jira/browse/HIVE-19504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472372#comment-16472372
]
Vineet Garg commented on HIVE-19504:
+1
> Change default value for hive.auto.convert.join.shuffle.max.size property
> -
>
> Key: HIVE-19504
> URL: https://issues.apache.org/jira/browse/HIVE-19504
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19504.patch
>
>
> The property default value is too low by mistake (10MB), it is missing three
> trailing zeros.
> {code}
> HIVECONVERTJOINMAXSHUFFLESIZE("hive.auto.convert.join.shuffle.max.size",
> 1000L,
>"If hive.auto.convert.join.noconditionaltask is off, this parameter
> does not take affect. \n" +
>"However, if it is on, and the predicted size of the larger input for
> a given join is greater \n" +
>"than this number, the join will not be converted to a dynamically
> partitioned hash join. \n" +
>"The value \"-1\" means no limit."),
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19489) Disable stats autogather for external tables
[ https://issues.apache.org/jira/browse/HIVE-19489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472377#comment-16472377 ] Jason Dere commented on HIVE-19489: --- So we've seen a lot of users who end up with very wrong stats because most of the data has been written to by external apps, which can be as bad for planning as no stats. The point of this would be to put the responsibility on the user to call ANALYZE TABLE to keep stats up-to-date, rather than assuming this will be taken care of by auto-gather. I'll try to follow up with Ashutosh on this one. You are right that LOAD DATA does not seem to fully perform stats auto-gather, though there still does appear to be some StatsWork that shows up in the plan during load. > Disable stats autogather for external tables > > > Key: HIVE-19489 > URL: https://issues.apache.org/jira/browse/HIVE-19489 > Project: Hive > Issue Type: Sub-task > Components: Statistics >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > Hive auto-gather of table statistics can result in incorrect generation of > stats (and the stats being marked as accurate) in the case of external tables > where the data is being written by external apps. > To avoid this issue, stats autogather will be disabled on external tables > when loading/inserting into a table with existing data, if > HIVE_DISABLE_UNSAFE_EXTERNALTABLE_OPERATIONS is enabled. In this situation, > users should rely on explicitly calling ANALYZE TABLE on their external > tables to make sure the stats are kept up-to-date. > Autogather of stats will still be allowed to occur on external tables in the > case of INSERT OVERWRITE or LOAD DATA OVERWRITE, since the existing data is > being removed and so the stats calculated on the inserted/loaded data should > be accurate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19497) SessionHiveMetaStoreClient.getTable should respect default catalog
[ https://issues.apache.org/jira/browse/HIVE-19497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472378#comment-16472378 ] Dongjoon Hyun commented on HIVE-19497: -- Thank you for review, [~alangates]. > SessionHiveMetaStoreClient.getTable should respect default catalog > -- > > Key: HIVE-19497 > URL: https://issues.apache.org/jira/browse/HIVE-19497 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Priority: Blocker > Attachments: > 0001-HIVE-19497-SessionHiveMetaStoreClient.getTable-shoul.patch > > > Currently, SessionHiveMetaStoreClient always use `DEFAULT_CATALOG_NAME` > during looking up the underlying tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19497) SessionHiveMetaStoreClient.getTable should respect default catalog
[ https://issues.apache.org/jira/browse/HIVE-19497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472382#comment-16472382 ] Vineet Garg commented on HIVE-19497: +1 for branch-3. I am not sure if attached patch is in correct format to be picked up by jenkins. > SessionHiveMetaStoreClient.getTable should respect default catalog > -- > > Key: HIVE-19497 > URL: https://issues.apache.org/jira/browse/HIVE-19497 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Priority: Blocker > Attachments: > 0001-HIVE-19497-SessionHiveMetaStoreClient.getTable-shoul.patch > > > Currently, SessionHiveMetaStoreClient always use `DEFAULT_CATALOG_NAME` > during looking up the underlying tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19041) Thrift deserialization of Partition objects should intern fields
[ https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472391#comment-16472391 ] Vihang Karajgaonkar commented on HIVE-19041: not sure why the patch couldn't be applied. The code compiles for me locally. > Thrift deserialization of Partition objects should intern fields > > > Key: HIVE-19041 > URL: https://issues.apache.org/jira/browse/HIVE-19041 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 2.3.2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-19041.01.patch, HIVE-19041.02.patch, > HIVE-19041.03.patch, HIVE-19041.04.patch > > > When a client is creating large number of partitions, the thrift objects are > deserialized into Partition objects. The read method of these objects does > not intern the inputformat, location, outputformat which cause large number > of duplicate Strings in the HMS memory. We should intern these objects while > deserialization to reduce memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19326) union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats (incorrect query results possible)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472400#comment-16472400
]
Zoltan Haindrich commented on HIVE-19326:
-
I've a "half-fix": it's fixed in most cases, but if {{hive.merge.tezfiles}} is
enabled the problem still occurs.
> union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats
> (incorrect query results possible)
>
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Blocker
> Fix For: 3.0.0
>
>
> Found when investigating the results change after converting tables to MM,
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 5
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 15
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles2
> numRows 20
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate,
> but BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect
> results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19326) union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats (incorrect query results possible)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19326:
Attachment: HIVE-19326.01wip01.patch
> union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats
> (incorrect query results possible)
>
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HIVE-19326.01wip01.patch
>
>
> Found when investigating the results change after converting tables to MM,
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 5
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 15
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles2
> numRows 20
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate,
> but BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect
> results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19326) union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats (incorrect query results possible)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19326:
Status: Patch Available (was: Open)
> union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats
> (incorrect query results possible)
>
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HIVE-19326.01wip01.patch
>
>
> Found when investigating the results change after converting tables to MM,
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 5
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 15
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles2
> numRows 20
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate,
> but BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect
> results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19326) union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats (incorrect query results possible)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472415#comment-16472415
]
Sergey Shelukhin commented on HIVE-19326:
-
Should we commit that and then do a follow-up?
> union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats
> (incorrect query results possible)
>
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HIVE-19326.01wip01.patch
>
>
> Found when investigating the results change after converting tables to MM,
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 5
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 15
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles2
> numRows 20
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate,
> but BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect
> results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19041) Thrift deserialization of Partition objects should intern fields
[
https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472418#comment-16472418
]
Vihang Karajgaonkar commented on HIVE-19041:
Looked more carefully and realized that the issue is not with
smart-apply-patch.sh but something went wrong with maven.
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(process-resource-bundles) on project hive-shims-scheduler: Execution
process-resource-bundles of goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed.
ConcurrentModificationException -> [Help 1]
{noformat}
So much for waiting 2 days for the precommit. The patch doesn't change any code
functionality and there is no test which checks for string interning as far as
I know. So precommit job for this particular job isn't going to test anything.
I am going to go ahead and merge the patch.
> Thrift deserialization of Partition objects should intern fields
>
>
> Key: HIVE-19041
> URL: https://issues.apache.org/jira/browse/HIVE-19041
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-19041.01.patch, HIVE-19041.02.patch,
> HIVE-19041.03.patch, HIVE-19041.04.patch
>
>
> When a client is creating large number of partitions, the thrift objects are
> deserialized into Partition objects. The read method of these objects does
> not intern the inputformat, location, outputformat which cause large number
> of duplicate Strings in the HMS memory. We should intern these objects while
> deserialization to reduce memory pressure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19326) union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats (incorrect query results possible)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472436#comment-16472436
]
Zoltan Haindrich commented on HIVE-19326:
-
well..I think we are better of with it; then without it :)
> union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats
> (incorrect query results possible)
>
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HIVE-19326.01wip01.patch
>
>
> Found when investigating the results change after converting tables to MM,
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 5
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 15
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles2
> numRows 20
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate,
> but BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect
> results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19481) sample10.q returns possibly wrong results for insert-only transactional table
[ https://issues.apache.org/jira/browse/HIVE-19481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472442#comment-16472442 ] Sergey Shelukhin commented on HIVE-19481: - [~steveyeom2017] are you saying HIVE-18910 broke tablesample? [~djaiswal] the patch for murmur causes the queries in sample10 to produce no results. Was that by design? I see the test itself was actually changed for some reason in that patch. Seems like sample should still return some rows (which it does if table is converted to MM). > sample10.q returns possibly wrong results for insert-only transactional table > - > > Key: HIVE-19481 > URL: https://issues.apache.org/jira/browse/HIVE-19481 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 3.1.0 > > > Ran "mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=sample10.q " after > changing the table to be > insert-only transactional. > The following queries returns couple of rows whereis no row results returns > for non-ACID table. > query: select ds, count(1) from srcpartbucket tablesample (bucket 2 out of 4 > on key) where ds is not null group by ds ORDER BY ds ASC > 2008-04-08 14 > 2008-04-09 14 > .. > query: select ds, count(1) from srcpartbucket tablesample (bucket 1 out of 2 > on key) where ds is not null group by ds ORDER BY ds ASC > 2008-04-08 4 > 2008-04-09 4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19481) sample10.q returns possibly wrong results for insert-only transactional table
[ https://issues.apache.org/jira/browse/HIVE-19481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472449#comment-16472449 ] Steve Yeom commented on HIVE-19481: --- Hey [~sershe]I talked with Deepak and Jason. Deepak will check a bit. > sample10.q returns possibly wrong results for insert-only transactional table > - > > Key: HIVE-19481 > URL: https://issues.apache.org/jira/browse/HIVE-19481 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 3.1.0 > > > Ran "mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=sample10.q " after > changing the table to be > insert-only transactional. > The following queries returns couple of rows whereis no row results returns > for non-ACID table. > query: select ds, count(1) from srcpartbucket tablesample (bucket 2 out of 4 > on key) where ds is not null group by ds ORDER BY ds ASC > 2008-04-08 14 > 2008-04-09 14 > .. > query: select ds, count(1) from srcpartbucket tablesample (bucket 1 out of 2 > on key) where ds is not null group by ds ORDER BY ds ASC > 2008-04-08 4 > 2008-04-09 4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19384) Vectorization: IfExprTimestamp* do not handle NULLs correctly
[
https://issues.apache.org/jira/browse/HIVE-19384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472470#comment-16472470
]
Hive QA commented on HIVE-19384:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922780/HIVE-19384.04.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 62 failed/errored test(s), 13591 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,mm_update.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,udf_sort_array_by_wrong2.q,
