[jira] [Updated] (HIVE-18832) Support change management for trashing data files from ACID tables.

2018-03-11 Thread anishek (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anishek updated HIVE-18832:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

committed to master 

> Support change management for trashing data files from ACID tables.
> ---
>
> Key: HIVE-18832
> URL: https://issues.apache.org/jira/browse/HIVE-18832
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18832.0.patch, HIVE-18832.1.patch, 
> HIVE-18832.2.patch
>
>
> Currently, cleaner process and DDL drop operations deletes the data files. 
> So, scope to support change management in source warehouse for ACID table 
> operations is given below.
> 1. Cleaner process deletes older files after compaction, aborted files etc. 
> Need to be archived to cmroot path.
> 2. DDL operations such as Drop table, partition already archive the deleted 
> files. Need to extend it for ACID tables too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18832) Support change management for trashing data files from ACID tables.

2018-03-11 Thread anishek (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anishek updated HIVE-18832:
---
Attachment: HIVE-18832.2.patch

> Support change management for trashing data files from ACID tables.
> ---
>
> Key: HIVE-18832
> URL: https://issues.apache.org/jira/browse/HIVE-18832
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18832.0.patch, HIVE-18832.1.patch, 
> HIVE-18832.2.patch
>
>
> Currently, cleaner process and DDL drop operations deletes the data files. 
> So, scope to support change management in source warehouse for ACID table 
> operations is given below.
> 1. Cleaner process deletes older files after compaction, aborted files etc. 
> Need to be archived to cmroot path.
> 2. DDL operations such as Drop table, partition already archive the deleted 
> files. Need to extend it for ACID tables too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18832) Support change management for trashing data files from ACID tables.

2018-03-11 Thread anishek (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394849#comment-16394849
 ] 

anishek commented on HIVE-18832:


added the patch with a file name change and adding license header.

> Support change management for trashing data files from ACID tables.
> ---
>
> Key: HIVE-18832
> URL: https://issues.apache.org/jira/browse/HIVE-18832
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18832.0.patch, HIVE-18832.1.patch, 
> HIVE-18832.2.patch
>
>
> Currently, cleaner process and DDL drop operations deletes the data files. 
> So, scope to support change management in source warehouse for ACID table 
> operations is given below.
> 1. Cleaner process deletes older files after compaction, aborted files etc. 
> Need to be archived to cmroot path.
> 2. DDL operations such as Drop table, partition already archive the deleted 
> files. Need to extend it for ACID tables too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18832) Support change management for trashing data files from ACID tables.

2018-03-11 Thread anishek (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394847#comment-16394847
 ] 

anishek commented on HIVE-18832:


* org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] 
: runs fine on local machine
* org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_varchar1] : runs 
fine on local machine 
* 
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]:
 runs fine on local machine


other failures are from older builds.

> Support change management for trashing data files from ACID tables.
> ---
>
> Key: HIVE-18832
> URL: https://issues.apache.org/jira/browse/HIVE-18832
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18832.0.patch, HIVE-18832.1.patch
>
>
> Currently, cleaner process and DDL drop operations deletes the data files. 
> So, scope to support change management in source warehouse for ACID table 
> operations is given below.
> 1. Cleaner process deletes older files after compaction, aborted files etc. 
> Need to be archived to cmroot path.
> 2. DDL operations such as Drop table, partition already archive the deleted 
> files. Need to extend it for ACID tables too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs

2018-03-11 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394826#comment-16394826
 ] 

Hive QA commented on HIVE-18533:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913962/HIVE-18533.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 29 failed/errored test(s), 12882 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,nopart_load.q,alter_table_non_partitioned_table_cascade.q,load_wrong_fileformat.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,udf_min.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,authorization_insert_noinspriv.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,msck_repair_3.q,udf_sort_array_by_wrong2.q,local_mapred_error_cache.q,alter_external_acid.q,mm_concatenate.q,authorization_fail_3.q,set_hiveconf_internal_variable0.q,udf_last_day_error_2.q,alter_table_constraint_invalid_ref.q,create_table_wrong_regex.q,describe_x

[jira] [Commented] (HIVE-18748) Rename table impacts the ACID behaviour as table names are not updated in meta-tables.

2018-03-11 Thread Sankar Hariappan (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394823#comment-16394823
 ] 

Sankar Hariappan commented on HIVE-18748:
-

Currently, we propose to disable rename table/partition operations if it 
involves ACID table.

> Rename table impacts the ACID behaviour as table names are not updated in 
> meta-tables.
> --
>
> Key: HIVE-18748
> URL: https://issues.apache.org/jira/browse/HIVE-18748
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, DDL
> Fix For: 3.0.0
>
>
> ACID implementation uses metatables such as TXN_COMPONENTS, 
> COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to 
> manage ACID operations.
> Per table write ID implementation (HIVE-18192) introduces couple of 
> metatables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids 
> allocated per table.
> Now, when we rename any tables, it is necessary to update the corresponding 
> table names in these metatables as well. Otherwise, ACID table operations 
> won't work properly.
> Since, this change is significant and have other side-effects, we propose to 
> disable rename tables on ACID tables until a fix is figured out.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18748) Rename table impacts the ACID behaviour as table names are not updated in meta-tables.

2018-03-11 Thread Sankar Hariappan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-18748:

Description: 
ACID implementation uses metatables such as TXN_COMPONENTS, 
COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to 
manage ACID operations.

Per table write ID implementation (HIVE-18192) introduces couple of metatables 
such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids allocated per 
table.

Now, when we rename any tables, it is necessary to update the corresponding 
table names in these metatables as well. Otherwise, ACID table operations won't 
work properly.

Since, this change is significant and have other side-effects, we propose to 
disable rename tables on ACID tables until a fix is figured out.

  was:
ACID implementation uses metatables such as TXN_COMPONENTS, 
COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to 
manage ACID operations.

Per table write ID implementation (HIVE-18192) introduces couple of metatables 
such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids allocated per 
table.

Now, when we rename any tables, it is necessary to update the corresponding 
table names in these metatables as well. Otherwise, ACID table operations won't 
work properly.

Since, this change is significant, we propose to disable rename tables on ACID 
tables until a fix is figured out.


> Rename table impacts the ACID behaviour as table names are not updated in 
> meta-tables.
> --
>
> Key: HIVE-18748
> URL: https://issues.apache.org/jira/browse/HIVE-18748
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, DDL
> Fix For: 3.0.0
>
>
> ACID implementation uses metatables such as TXN_COMPONENTS, 
> COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to 
> manage ACID operations.
> Per table write ID implementation (HIVE-18192) introduces couple of 
> metatables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids 
> allocated per table.
> Now, when we rename any tables, it is necessary to update the corresponding 
> table names in these metatables as well. Otherwise, ACID table operations 
> won't work properly.
> Since, this change is significant and have other side-effects, we propose to 
> disable rename tables on ACID tables until a fix is figured out.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18748) Rename table impacts the ACID behaviour as table names are not updated in meta-tables.

2018-03-11 Thread Sankar Hariappan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-18748:

Description: 
ACID implementation uses metatables such as TXN_COMPONENTS, 
COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to 
manage ACID operations.

Per table write ID implementation (HIVE-18192) introduces couple of metatables 
such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids allocated per 
table.

Now, when we rename any tables, it is necessary to update the corresponding 
table names in these metatables as well. Otherwise, ACID table operations won't 
work properly.

Since, this change is significant, we propose to disable rename tables on ACID 
tables until a fix is figured out.

  was:
Per table write ID implementation (HIVE-18192) introduces couple of metatables 
such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids allocated per 
table.

Now, when we rename any tables, it is necessary to update the corresponding 
table names in these table as well. Otherwise, ACID table operations won't work 
properly.


> Rename table impacts the ACID behaviour as table names are not updated in 
> meta-tables.
> --
>
> Key: HIVE-18748
> URL: https://issues.apache.org/jira/browse/HIVE-18748
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, DDL
> Fix For: 3.0.0
>
>
> ACID implementation uses metatables such as TXN_COMPONENTS, 
> COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to 
> manage ACID operations.
> Per table write ID implementation (HIVE-18192) introduces couple of 
> metatables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids 
> allocated per table.
> Now, when we rename any tables, it is necessary to update the corresponding 
> table names in these metatables as well. Otherwise, ACID table operations 
> won't work properly.
> Since, this change is significant, we propose to disable rename tables on 
> ACID tables until a fix is figured out.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18748) Rename table impacts the ACID behaviour as table names are not updated in meta-tables.

2018-03-11 Thread Sankar Hariappan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-18748:

Summary: Rename table impacts the ACID behaviour as table names are not 
updated in meta-tables.  (was: Rename table affects ACID table behaviour as 
table names are not corrected in meta-tables.)

> Rename table impacts the ACID behaviour as table names are not updated in 
> meta-tables.
> --
>
> Key: HIVE-18748
> URL: https://issues.apache.org/jira/browse/HIVE-18748
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, DDL
> Fix For: 3.0.0
>
>
> Per table write ID implementation (HIVE-18192) introduces couple of 
> metatables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids 
> allocated per table.
> Now, when we rename any tables, it is necessary to update the corresponding 
> table names in these table as well. Otherwise, ACID table operations won't 
> work properly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18748) Rename table affects ACID table behaviour as table names are not corrected in meta-tables.

2018-03-11 Thread Sankar Hariappan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-18748:

Summary: Rename table affects ACID table behaviour as table names are not 
corrected in meta-tables.  (was: Rename table should update the table names in 
NEXT_WRITE_ID and TXN_TO_WRITE_ID tables. )

> Rename table affects ACID table behaviour as table names are not corrected in 
> meta-tables.
> --
>
> Key: HIVE-18748
> URL: https://issues.apache.org/jira/browse/HIVE-18748
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, DDL
> Fix For: 3.0.0
>
>
> Per table write ID implementation (HIVE-18192) introduces couple of 
> metatables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids 
> allocated per table.
> Now, when we rename any tables, it is necessary to update the corresponding 
> table names in these table as well. Otherwise, ACID table operations won't 
> work properly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18860) fix TestAcidOnTez#testGetSplitsLocks

2018-03-11 Thread Sankar Hariappan (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394820#comment-16394820
 ] 

Sankar Hariappan commented on HIVE-18860:
-

HIVE-18751 is fixed now for IOException. So, now we are getting the NPE as 
before HIVE-18192 merge. 

> fix TestAcidOnTez#testGetSplitsLocks
> 
>
> Key: HIVE-18860
> URL: https://issues.apache.org/jira/browse/HIVE-18860
> Project: Hive
>  Issue Type: Bug
>  Components: Test, Transactions
>Reporter: Zoltan Haindrich
>Priority: Major
>
> it seems to me that HIVE-18665 patch have broken this test 
> https://travis-ci.org/kgyrtkirk/hive/builds/345287889



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18751) ACID table scan through get_splits UDF doesn't receive ValidWriteIdList configuration.

2018-03-11 Thread Sankar Hariappan (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394815#comment-16394815
 ] 

Sankar Hariappan commented on HIVE-18751:
-

Thanks for the review [~anishek]!

Test failures are irrelevant to the patch and it is committed to master!

> ACID table scan through get_splits UDF doesn't receive ValidWriteIdList 
> configuration.
> --
>
> Key: HIVE-18751
> URL: https://issues.apache.org/jira/browse/HIVE-18751
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, UDF, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18751.01.patch
>
>
> Per table write ID (HIVE-18192) have replaced global transaction ID with 
> write ID to version data files in ACID/MM tables,
> To ensure snapshot isolation, need to generate ValidWriteIdList for the given 
> txn/table and use it when scan the ACID/MM tables.
> In case of get_splits UDF which runs on ACID table scan query won't receive 
> it properly through configuration (hive.txn.tables.valid.writeids) and hence 
> throws exception. 
> TestAcidOnTez.testGetSplitsLocks is the test failing for the same. Need to 
> fix it.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18751) ACID table scan through get_splits UDF doesn't receive ValidWriteIdList configuration.

2018-03-11 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394816#comment-16394816
 ] 

ASF GitHub Bot commented on HIVE-18751:
---

Github user sankarh closed the pull request at:

https://github.com/apache/hive/pull/317


> ACID table scan through get_splits UDF doesn't receive ValidWriteIdList 
> configuration.
> --
>
> Key: HIVE-18751
> URL: https://issues.apache.org/jira/browse/HIVE-18751
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, UDF, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18751.01.patch
>
>
> Per table write ID (HIVE-18192) have replaced global transaction ID with 
> write ID to version data files in ACID/MM tables,
> To ensure snapshot isolation, need to generate ValidWriteIdList for the given 
> txn/table and use it when scan the ACID/MM tables.
> In case of get_splits UDF which runs on ACID table scan query won't receive 
> it properly through configuration (hive.txn.tables.valid.writeids) and hence 
> throws exception. 
> TestAcidOnTez.testGetSplitsLocks is the test failing for the same. Need to 
> fix it.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18751) ACID table scan through get_splits UDF doesn't receive ValidWriteIdList configuration.

2018-03-11 Thread Sankar Hariappan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-18751:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> ACID table scan through get_splits UDF doesn't receive ValidWriteIdList 
> configuration.
> --
>
> Key: HIVE-18751
> URL: https://issues.apache.org/jira/browse/HIVE-18751
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, UDF, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18751.01.patch
>
>
> Per table write ID (HIVE-18192) have replaced global transaction ID with 
> write ID to version data files in ACID/MM tables,
> To ensure snapshot isolation, need to generate ValidWriteIdList for the given 
> txn/table and use it when scan the ACID/MM tables.
> In case of get_splits UDF which runs on ACID table scan query won't receive 
> it properly through configuration (hive.txn.tables.valid.writeids) and hence 
> throws exception. 
> TestAcidOnTez.testGetSplitsLocks is the test failing for the same. Need to 
> fix it.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs

2018-03-11 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394813#comment-16394813
 ] 

Hive QA commented on HIVE-18533:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m  
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  5m 
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m  
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
15s{color} | {color:red} common: The patch generated 1 new + 426 unchanged - 0 
fixed = 427 total (was 426) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
37s{color} | {color:red} root: The patch generated 39 new + 448 unchanged - 15 
fixed = 487 total (was 463) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m  
9s{color} | {color:red} spark-client: The patch generated 38 new + 22 unchanged 
- 15 fixed = 60 total (was 37) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 13 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  5m 
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
18s{color} | {color:red} The patch generated 50 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 31s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9597/dev-support/hive-personality.sh
 |
| git revision | master / e213c4c |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9597/yetus/diff-checkstyle-common.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9597/yetus/diff-checkstyle-root.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9597/yetus/diff-checkstyle-spark-client.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9597/yetus/whitespace-tabs.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9597/yetus/patch-asflicense-problems.txt
 |
| modules | C: common . itests/qtest-spark spark-client U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9597/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Add option to use InProcessLauncher to submit spark jobs
> 
>
> Key: HIVE-18533
> URL: https://issues.apache.org/jira/browse/HIVE-18533
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachm

[jira] [Commented] (HIVE-18832) Support change management for trashing data files from ACID tables.

2018-03-11 Thread Sankar Hariappan (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394810#comment-16394810
 ] 

Sankar Hariappan commented on HIVE-18832:
-

+1

> Support change management for trashing data files from ACID tables.
> ---
>
> Key: HIVE-18832
> URL: https://issues.apache.org/jira/browse/HIVE-18832
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18832.0.patch, HIVE-18832.1.patch
>
>
> Currently, cleaner process and DDL drop operations deletes the data files. 
> So, scope to support change management in source warehouse for ACID table 
> operations is given below.
> 1. Cleaner process deletes older files after compaction, aborted files etc. 
> Need to be archived to cmroot path.
> 2. DDL operations such as Drop table, partition already archive the deleted 
> files. Need to extend it for ACID tables too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs

2018-03-11 Thread Sahil Takiar (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-18533:

Status: Patch Available  (was: Open)

> Add option to use InProcessLauncher to submit spark jobs
> 
>
> Key: HIVE-18533
> URL: https://issues.apache.org/jira/browse/HIVE-18533
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18533.1.patch
>
>
> See discussion in HIVE-16484 for details.
> I think this will help with reducing the amount of time it takes to open a 
> HoS session + debuggability (no need launch a separate process to run a Spark 
> app).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs

2018-03-11 Thread Sahil Takiar (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394793#comment-16394793
 ] 

Sahil Takiar commented on HIVE-18533:
-

Attaching WIP patch with the {{SparkLauncher}} option enabled by default, but 
just for testing purposes.

> Add option to use InProcessLauncher to submit spark jobs
> 
>
> Key: HIVE-18533
> URL: https://issues.apache.org/jira/browse/HIVE-18533
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18533.1.patch
>
>
> See discussion in HIVE-16484 for details.
> I think this will help with reducing the amount of time it takes to open a 
> HoS session + debuggability (no need launch a separate process to run a Spark 
> app).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs

2018-03-11 Thread Sahil Takiar (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-18533:

Attachment: HIVE-18533.1.patch

> Add option to use InProcessLauncher to submit spark jobs
> 
>
> Key: HIVE-18533
> URL: https://issues.apache.org/jira/browse/HIVE-18533
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18533.1.patch
>
>
> See discussion in HIVE-16484 for details.
> I think this will help with reducing the amount of time it takes to open a 
> HoS session + debuggability (no need launch a separate process to run a Spark 
> app).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18751) ACID table scan through get_splits UDF doesn't receive ValidWriteIdList configuration.

2018-03-11 Thread anishek (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394792#comment-16394792
 ] 

anishek commented on HIVE-18751:


+ 1

> ACID table scan through get_splits UDF doesn't receive ValidWriteIdList 
> configuration.
> --
>
> Key: HIVE-18751
> URL: https://issues.apache.org/jira/browse/HIVE-18751
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, UDF, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18751.01.patch
>
>
> Per table write ID (HIVE-18192) have replaced global transaction ID with 
> write ID to version data files in ACID/MM tables,
> To ensure snapshot isolation, need to generate ValidWriteIdList for the given 
> txn/table and use it when scan the ACID/MM tables.
> In case of get_splits UDF which runs on ACID table scan query won't receive 
> it properly through configuration (hive.txn.tables.valid.writeids) and hence 
> throws exception. 
> TestAcidOnTez.testGetSplitsLocks is the test failing for the same. Need to 
> fix it.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18859) Incorrect handling of thrift metastore exceptions

2018-03-11 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394765#comment-16394765
 ] 

Hive QA commented on HIVE-18859:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913957/HIVE-18859.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9596/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9596/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9596/

Messages:
{noformat}
 This message was trimmed, see log for full details 
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.8.v20160314/jetty-servlet-9.3.8.v20160314.jar(org/eclipse/jetty/servlet/FilterHolder.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.8.v20160314/jetty-servlet-9.3.8.v20160314.jar(org/eclipse/jetty/servlet/ServletContextHandler.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.8.v20160314/jetty-servlet-9.3.8.v20160314.jar(org/eclipse/jetty/servlet/ServletHolder.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-xml/9.3.8.v20160314/jetty-xml-9.3.8.v20160314.jar(org/eclipse/jetty/xml/XmlConfiguration.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/slf4j/jul-to-slf4j/1.7.10/jul-to-slf4j-1.7.10.jar(org/slf4j/bridge/SLF4JBridgeHandler.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/DispatcherType.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/Filter.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/FilterChain.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/FilterConfig.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/ServletException.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/ServletRequest.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/ServletResponse.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/annotation/WebFilter.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/http/HttpServletRequest.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/http/HttpServletResponse.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/classification/target/hive-classification-3.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceAudience$LimitedPrivate.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/classification/target/hive-classification-3.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceStability$Unstable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/ByteArrayOutputStream.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/OutputStream.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Closeable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/AutoCloseable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Flushable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(javax/xml/bind/annotation/XmlRootElement.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/commons/commons-exec/1.1/commons-exec-1.1.jar(org/apache/commons/exec/ExecuteException.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/security/PrivilegedExceptionAction.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/ExecutionException.class)]]
[loading 
ZipFileIndexFileObject[/usr/

[jira] [Updated] (HIVE-18859) Incorrect handling of thrift metastore exceptions

2018-03-11 Thread Ganesha Shreedhara (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ganesha Shreedhara updated HIVE-18859:
--
Attachment: (was: HIVE-18859.2.patch)

> Incorrect handling of thrift metastore exceptions
> -
>
> Key: HIVE-18859
> URL: https://issues.apache.org/jira/browse/HIVE-18859
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.0, 2.1.1
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-18859.patch
>
>
> Currently any run time exception thrown in thrift metastore during the 
> following operations is not getting sent to hive execution engine.
>  * grant/revoke role
>  * grant/revoke privileges
>  * create role
> This is because ThriftHiveMetastore just handles MetaException and throws 
> TException during the processing of these requests.  So, the command just 
> fails at thrift metastore end when there is run time exception (Exception can 
> be seen in metastore log) but the hive execution engine will keep on waiting 
> for the response from thrift metatstore.
>  
> Steps to reproduce this problem :
> Launch thrift metastore
> Launch hive cli by passing --hiveconf 
> hive.metastore.uris=thrift://127.0.0.1:1 (pass the thrift metatstore host 
> and port)
> Execute the following commands:
>  # set role admin
>  # create role test; (succeeds)
>  # create role test; ( hive version 2.1.1 : command is stuck, waiting for the 
> response from thrift metastore; hive version 1.2.1: command fails with 
> exception as null) 
>  
> I have uploaded the patch which has the fix in which I am handling the 
> checked exceptions in MetaException and throwing unchecked exceptions using 
> TException which fixes the problem. Please review and suggest if there is a 
> better way of handling this issue.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18859) Incorrect handling of thrift metastore exceptions

2018-03-11 Thread Ganesha Shreedhara (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ganesha Shreedhara updated HIVE-18859:
--
Attachment: HIVE-18859.patch

> Incorrect handling of thrift metastore exceptions
> -
>
> Key: HIVE-18859
> URL: https://issues.apache.org/jira/browse/HIVE-18859
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.0, 2.1.1
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-18859.patch
>
>
> Currently any run time exception thrown in thrift metastore during the 
> following operations is not getting sent to hive execution engine.
>  * grant/revoke role
>  * grant/revoke privileges
>  * create role
> This is because ThriftHiveMetastore just handles MetaException and throws 
> TException during the processing of these requests.  So, the command just 
> fails at thrift metastore end when there is run time exception (Exception can 
> be seen in metastore log) but the hive execution engine will keep on waiting 
> for the response from thrift metatstore.
>  
> Steps to reproduce this problem :
> Launch thrift metastore
> Launch hive cli by passing --hiveconf 
> hive.metastore.uris=thrift://127.0.0.1:1 (pass the thrift metatstore host 
> and port)
> Execute the following commands:
>  # set role admin
>  # create role test; (succeeds)
>  # create role test; ( hive version 2.1.1 : command is stuck, waiting for the 
> response from thrift metastore; hive version 1.2.1: command fails with 
> exception as null) 
>  
> I have uploaded the patch which has the fix in which I am handling the 
> checked exceptions in MetaException and throwing unchecked exceptions using 
> TException which fixes the problem. Please review and suggest if there is a 
> better way of handling this issue.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18859) Incorrect handling of thrift metastore exceptions

2018-03-11 Thread Ganesha Shreedhara (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ganesha Shreedhara updated HIVE-18859:
--
Status: Open  (was: Patch Available)

> Incorrect handling of thrift metastore exceptions
> -
>
> Key: HIVE-18859
> URL: https://issues.apache.org/jira/browse/HIVE-18859
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.1, 1.2.0
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-18859.patch
>
>
> Currently any run time exception thrown in thrift metastore during the 
> following operations is not getting sent to hive execution engine.
>  * grant/revoke role
>  * grant/revoke privileges
>  * create role
> This is because ThriftHiveMetastore just handles MetaException and throws 
> TException during the processing of these requests.  So, the command just 
> fails at thrift metastore end when there is run time exception (Exception can 
> be seen in metastore log) but the hive execution engine will keep on waiting 
> for the response from thrift metatstore.
>  
> Steps to reproduce this problem :
> Launch thrift metastore
> Launch hive cli by passing --hiveconf 
> hive.metastore.uris=thrift://127.0.0.1:1 (pass the thrift metatstore host 
> and port)
> Execute the following commands:
>  # set role admin
>  # create role test; (succeeds)
>  # create role test; ( hive version 2.1.1 : command is stuck, waiting for the 
> response from thrift metastore; hive version 1.2.1: command fails with 
> exception as null) 
>  
> I have uploaded the patch which has the fix in which I am handling the 
> checked exceptions in MetaException and throwing unchecked exceptions using 
> TException which fixes the problem. Please review and suggest if there is a 
> better way of handling this issue.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18859) Incorrect handling of thrift metastore exceptions

2018-03-11 Thread Ganesha Shreedhara (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ganesha Shreedhara updated HIVE-18859:
--
Status: Patch Available  (was: Open)

> Incorrect handling of thrift metastore exceptions
> -
>
> Key: HIVE-18859
> URL: https://issues.apache.org/jira/browse/HIVE-18859
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.1, 1.2.0
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-18859.patch
>
>
> Currently any run time exception thrown in thrift metastore during the 
> following operations is not getting sent to hive execution engine.
>  * grant/revoke role
>  * grant/revoke privileges
>  * create role
> This is because ThriftHiveMetastore just handles MetaException and throws 
> TException during the processing of these requests.  So, the command just 
> fails at thrift metastore end when there is run time exception (Exception can 
> be seen in metastore log) but the hive execution engine will keep on waiting 
> for the response from thrift metatstore.
>  
> Steps to reproduce this problem :
> Launch thrift metastore
> Launch hive cli by passing --hiveconf 
> hive.metastore.uris=thrift://127.0.0.1:1 (pass the thrift metatstore host 
> and port)
> Execute the following commands:
>  # set role admin
>  # create role test; (succeeds)
>  # create role test; ( hive version 2.1.1 : command is stuck, waiting for the 
> response from thrift metastore; hive version 1.2.1: command fails with 
> exception as null) 
>  
> I have uploaded the patch which has the fix in which I am handling the 
> checked exceptions in MetaException and throwing unchecked exceptions using 
> TException which fixes the problem. Please review and suggest if there is a 
> better way of handling this issue.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18859) Incorrect handling of thrift metastore exceptions

2018-03-11 Thread Ganesha Shreedhara (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ganesha Shreedhara updated HIVE-18859:
--
Attachment: (was: HIVE-18859.patch)

> Incorrect handling of thrift metastore exceptions
> -
>
> Key: HIVE-18859
> URL: https://issues.apache.org/jira/browse/HIVE-18859
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.0, 2.1.1
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-18859.2.patch
>
>
> Currently any run time exception thrown in thrift metastore during the 
> following operations is not getting sent to hive execution engine.
>  * grant/revoke role
>  * grant/revoke privileges
>  * create role
> This is because ThriftHiveMetastore just handles MetaException and throws 
> TException during the processing of these requests.  So, the command just 
> fails at thrift metastore end when there is run time exception (Exception can 
> be seen in metastore log) but the hive execution engine will keep on waiting 
> for the response from thrift metatstore.
>  
> Steps to reproduce this problem :
> Launch thrift metastore
> Launch hive cli by passing --hiveconf 
> hive.metastore.uris=thrift://127.0.0.1:1 (pass the thrift metatstore host 
> and port)
> Execute the following commands:
>  # set role admin
>  # create role test; (succeeds)
>  # create role test; ( hive version 2.1.1 : command is stuck, waiting for the 
> response from thrift metastore; hive version 1.2.1: command fails with 
> exception as null) 
>  
> I have uploaded the patch which has the fix in which I am handling the 
> checked exceptions in MetaException and throwing unchecked exceptions using 
> TException which fixes the problem. Please review and suggest if there is a 
> better way of handling this issue.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-17178) Spark Partition Pruning Sink Operator can't target multiple Works

2018-03-11 Thread Rui Li (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-17178:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master. Thanks [~stakiar] for the review.

> Spark Partition Pruning Sink Operator can't target multiple Works
> -
>
> Key: HIVE-17178
> URL: https://issues.apache.org/jira/browse/HIVE-17178
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Rui Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-17178.1.patch, HIVE-17178.2.patch, 
> HIVE-17178.3.patch, HIVE-17178.4.patch, HIVE-17178.5.patch, HIVE-17178.6.patch
>
>
> A Spark Partition Pruning Sink Operator cannot be used to target multiple Map 
> Work objects. The entire DPP subtree (SEL-GBY-SPARKPRUNINGSINK) is duplicated 
> if a single table needs to be used to target multiple Map Works.
> The following query shows the issue:
> {code}
> set hive.spark.dynamic.partition.pruning=true;
> set hive.auto.convert.join=true;
> create table part_table_1 (col int) partitioned by (part_col int);
> create table part_table_2 (col int) partitioned by (part_col int);
> create table regular_table (col int);
> insert into table regular_table values (1);
> alter table part_table_1 add partition (part_col=1);
> insert into table part_table_1 partition (part_col=1) values (1), (2), (3), 
> (4);
> alter table part_table_1 add partition (part_col=2);
> insert into table part_table_1 partition (part_col=2) values (1), (2), (3), 
> (4);
> alter table part_table_2 add partition (part_col=1);
> insert into table part_table_2 partition (part_col=1) values (1), (2), (3), 
> (4);
> alter table part_table_2 add partition (part_col=2);
> insert into table part_table_2 partition (part_col=2) values (1), (2), (3), 
> (4);
> explain select * from regular_table, part_table_1, part_table_2 where 
> regular_table.col = part_table_1.part_col and regular_table.col = 
> part_table_2.part_col;
> {code}
> The explain plan is
> {code}
> STAGE DEPENDENCIES:
>   Stage-2 is a root stage
>   Stage-1 depends on stages: Stage-2
>   Stage-0 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-2
> Spark
>  A masked pattern was here 
>   Vertices:
> Map 1 
> Map Operator Tree:
> TableScan
>   alias: regular_table
>   Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE 
> Column stats: NONE
>   Filter Operator
> predicate: col is not null (type: boolean)
> Statistics: Num rows: 1 Data size: 1 Basic stats: 
> COMPLETE Column stats: NONE
> Select Operator
>   expressions: col (type: int)
>   outputColumnNames: _col0
>   Statistics: Num rows: 1 Data size: 1 Basic stats: 
> COMPLETE Column stats: NONE
>   Spark HashTable Sink Operator
> keys:
>   0 _col0 (type: int)
>   1 _col1 (type: int)
>   2 _col1 (type: int)
>   Select Operator
> expressions: _col0 (type: int)
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 1 Basic stats: 
> COMPLETE Column stats: NONE
> Group By Operator
>   keys: _col0 (type: int)
>   mode: hash
>   outputColumnNames: _col0
>   Statistics: Num rows: 1 Data size: 1 Basic stats: 
> COMPLETE Column stats: NONE
>   Spark Partition Pruning Sink Operator
> partition key expr: part_col
> Statistics: Num rows: 1 Data size: 1 Basic stats: 
> COMPLETE Column stats: NONE
> target column name: part_col
> target work: Map 2
>   Select Operator
> expressions: _col0 (type: int)
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 1 Basic stats: 
> COMPLETE Column stats: NONE
> Group By Operator
>   keys: _col0 (type: int)
>   mode: hash
>   outputColumnNames: _col0
>   Statistics: Num rows: 1 Data size: 1 Basic stats: 
> COMPLETE Column stats: NONE
>   Spark Partition Pruning Sink Operator
> partition key expr: part

[jira] [Commented] (HIVE-18675) make HIVE_LOCKS.HL_TXNID NOT NULL

2018-03-11 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394696#comment-16394696
 ] 

Hive QA commented on HIVE-18675:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913948/HIVE-18675.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 13350 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursion.q,merge_constraint_notnull.q,load_part_nospec.q,clusterbyorderby.q,orc_type_promotion2.q,ctas_noperm_loc.q,udf_instr_wrong_args_len.q,invalid_create_tbl2.q,part_col_complex_type.q,authorization_drop_db_empty.q,smb_mapjoin_14.q,subquery_scalar_multi_rows.q,alter_partition_coltype_2columns.q,subquery_corr_in_agg.q,insert_overwrite_notnull_constraint.q,authorization_show_grant_otheruser_wtab.q,regex_col_groupby.q,udaf_collect_set_unsupported.q,ptf_negative_DuplicateWindowAlias.q,exim_22_export_authfail.q,udf_likeany_wrong1.q,groupby_key.q,ambiguous_col.q,groupby3_multi_distinct.q,authorization_alter_drop_ptn.q,invalid_cast_from_binary_5.q,show_create_table_does_not_exist.q,invalid_select_column.q,exim_20_managed_location_over_existing.q,interval_3.q,authorization_compile.q,join35.q,merge_negative_3.q,udf_concat_ws_wrong3.q,create_or_replace_view8.q,create_external_with_notnull_constraint.q,split_sample_out_of_range.q,materialized_view_no_transactional_rewrite.q,authorization_show_grant_otherrole.q,create_with_constraints_duplicate_name.q,invalid_stddev_samp_syntax.q,authorization_view_disable_cbo_7.q,autolocal1.q,avro_non_nullable_union.q,load_orc_negative_part.q,drop_view_failure1.q,columnstats_partlvl_invalid_values_autogather.q,exim_13_nonnative_import.q,alter_table_wrong_regex.q,add_partition_with_whitelist.q,udf_next_day_error_2.q,authorization_select.q,udf_trunc_error2.q,authorization_view_7.q,udf_format_number_wrong5.q,touch2.q,exim_03_nonpart_noncompat_colschema.q,orc_type_promotion1.q,lateral_view_alias.q,show_tables_bad_db1.q,unset_table_property.q,alter_non_native.q,nvl_mismatch_type.q,load_orc_negative3.q,authorization_create_role_no_admin.q,invalid_distinct1.q,authorization_grant_server.q,orc_type_promotion3_acid.q,show_tables_bad1.q,macro_unused_parameter.q,drop_invalid_constraint3.q,drop_partition_filter_failure.q,char_pad_convert_fail3.q,exim_23_import_exist_authfail.q,drop_invalid_constraint4.q,authorization_create_macro1.q,archive1.q,subquery_multiple_cols_in_select.q,change_hive_hdfs_session_path.q,udf_trunc_error3.q,invalid_variance_syntax.q,authorization_truncate_2.q,invalid_avg_syntax.q,invalid_select_column_with_tablename.q,mm_truncate_cols.q,groupby_grouping_sets1.q,druid_location.q,groupby2_multi_distinct.q,authorization_sba_drop_table.q,dynamic_partitions_with_whitelist.q,compare_string_bigint_2.q,udf_greatest_error_2.q,authorization_view_6.q,show_tablestatus.q,duplicate_alias_in_transform_schema.q,create_with_fk_uk_same_tab.q,udtf_not_supported3.q,alter_table_constraint_invalid_fk_col2.q,udtf_not_supported1.q,dbtxnmgr_notableunlock.q,ptf_negative_InvalidValueBoundary.q,alter_table_constraint_duplicate_pk.q,udf_printf_wrong4.q,create_view_failure9.q,udf_elt_wrong_type.q,selectDistinctStarNeg_1.q,invalid_mapjoin1.q,load_stored_as_dirs.q,input1.q,udf_sort_array_wrong1.q,invalid_distinct2.q,invalid_select_fn.q,authorization_role_grant_otherrole.q,archive4.q,load_nonpart_authfail.q,recursive_view.q,authorization_view_disable_cbo_1.q,desc_failure4.q,create_not_acid.q,udf_sort_array_wrong3.q,char_pad_convert_fail0.q,udf_map_values_arg_type.q,alter_view_failure6_2.q,alter_partition_change_col_nonexist.q,update_non_acid_table.q,authorization_view_disable_cbo_5.q,ct_noperm_loc.q,interval_1.q,authorization_show_grant_otheruser_all.q,authorization_view_2.q,show_tables_bad2.q,groupby_rollup2.q,truncate_column_seqfile.q,create_view_failure5.q,authorization_create_view.q,ptf_window_boundaries.q,ctasnullcol.q,input_part0_neg_2.q,create_or_replace_view1.q,udf_max.q,exim_01_nonpart_over_loaded.q,msck_repair_1.q,orc_change_fileformat_acid.q,udf_nonexistent_resource.q,exim_19_external_over_existing.q,serde_regex2.q,msck_repair_2.q,exim_06_nonpart_noncompat_storage.q,illegal_partition_type4.q,udf_sort_array_by_wrong1.q,create_or_replace_view5.q,windowing_leadlag_in_udaf.q,avro_decimal.q,materialized_view_update.

[jira] [Commented] (HIVE-18675) make HIVE_LOCKS.HL_TXNID NOT NULL

2018-03-11 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394693#comment-16394693
 ] 

Hive QA commented on HIVE-18675:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m  
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 12m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m  
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 12m 
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  5m 
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 36s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9595/dev-support/hive-personality.sh
 |
| git revision | master / dd0bc33 |
| Default Java | 1.8.0_111 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9595/yetus/whitespace-tabs.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9595/yetus/patch-asflicense-problems.txt
 |
| modules | C: . metastore standalone-metastore U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9595/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> make HIVE_LOCKS.HL_TXNID NOT NULL
> -
>
> Key: HIVE-18675
> URL: https://issues.apache.org/jira/browse/HIVE-18675
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Kryvenko Igor
>Priority: Major
> Attachments: HIVE-18675.01.patch
>
>
> In Hive 3.0 all statements that may need locks run in a transaction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin

2018-03-11 Thread Matt McCline (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394676#comment-16394676
 ] 

Matt McCline commented on HIVE-18908:
-

Patch #2 is still a Work-In-Progress.

> Add support for FULL OUTER JOIN to MapJoin
> --
>
> Key: HIVE-18908
> URL: https://issues.apache.org/jira/browse/HIVE-18908
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch
>
>
> Currently, we do not support FULL OUTER JOIN in MapJoin.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin

2018-03-11 Thread Matt McCline (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-18908:

Attachment: HIVE-18908.02.patch

> Add support for FULL OUTER JOIN to MapJoin
> --
>
> Key: HIVE-18908
> URL: https://issues.apache.org/jira/browse/HIVE-18908
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch
>
>
> Currently, we do not support FULL OUTER JOIN in MapJoin.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-18930) Fix ppd_join5 test

2018-03-11 Thread Kryvenko Igor (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kryvenko Igor reassigned HIVE-18930:



> Fix ppd_join5 test
> --
>
> Key: HIVE-18930
> URL: https://issues.apache.org/jira/browse/HIVE-18930
> Project: Hive
>  Issue Type: Bug
>  Components: Test
>Reporter: Kryvenko Igor
>Assignee: Kryvenko Igor
>Priority: Major
>
> It seems that HIVE-18068 have broken this test.
> Log from Jenkins
> {code:java}
>  type="java.lang.AssertionError">
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18675) make HIVE_LOCKS.HL_TXNID NOT NULL

2018-03-11 Thread Kryvenko Igor (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kryvenko Igor updated HIVE-18675:
-
Attachment: HIVE-18675.01.patch

> make HIVE_LOCKS.HL_TXNID NOT NULL
> -
>
> Key: HIVE-18675
> URL: https://issues.apache.org/jira/browse/HIVE-18675
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Kryvenko Igor
>Priority: Major
> Attachments: HIVE-18675.01.patch
>
>
> In Hive 3.0 all statements that may need locks run in a transaction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18675) make HIVE_LOCKS.HL_TXNID NOT NULL

2018-03-11 Thread Kryvenko Igor (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kryvenko Igor updated HIVE-18675:
-
Status: Patch Available  (was: Open)

> make HIVE_LOCKS.HL_TXNID NOT NULL
> -
>
> Key: HIVE-18675
> URL: https://issues.apache.org/jira/browse/HIVE-18675
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Kryvenko Igor
>Priority: Major
> Attachments: HIVE-18675.01.patch
>
>
> In Hive 3.0 all statements that may need locks run in a transaction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-18675) make HIVE_LOCKS.HL_TXNID NOT NULL

2018-03-11 Thread Kryvenko Igor (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kryvenko Igor reassigned HIVE-18675:


Assignee: Kryvenko Igor

> make HIVE_LOCKS.HL_TXNID NOT NULL
> -
>
> Key: HIVE-18675
> URL: https://issues.apache.org/jira/browse/HIVE-18675
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Kryvenko Igor
>Priority: Major
>
> In Hive 3.0 all statements that may need locks run in a transaction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18560) qtests: QTestUtil refactor/split - QOutProcessor

2018-03-11 Thread Zoltan Haindrich (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394598#comment-16394598
 ] 

Zoltan Haindrich commented on HIVE-18560:
-

+1

> qtests: QTestUtil refactor/split - QOutProcessor
> 
>
> Key: HIVE-18560
> URL: https://issues.apache.org/jira/browse/HIVE-18560
> Project: Hive
>  Issue Type: Task
>  Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18560.01.patch, HIVE-18560.02.patch
>
>
> QTestUtil needs some refactoring, as it has become a large, hard-to-maintain, 
> hard-to-test class with a bunch of responsibilities:
>  * running qtests
>  * parsing/evaluating results
>  * masking/processing results
> First, I would extract a processing class from it (e.g. QOutProcessor or 
> something like that), which should be capable of being tested easily, and 
> contains all the logic what's needed to make the final q.out results from the 
> original q.out file. 
> This processing class shall remain untouched in case the testing 
> infrastructure (QTestUtil) is replaced or changed in some ways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18560) qtests: QTestUtil refactor/split - QOutProcessor

2018-03-11 Thread Laszlo Bodor (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394591#comment-16394591
 ] 

Laszlo Bodor commented on HIVE-18560:
-

Test failures are unrelated. 

[~kgyrtkirk] : could you review please?

> qtests: QTestUtil refactor/split - QOutProcessor
> 
>
> Key: HIVE-18560
> URL: https://issues.apache.org/jira/browse/HIVE-18560
> Project: Hive
>  Issue Type: Task
>  Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18560.01.patch, HIVE-18560.02.patch
>
>
> QTestUtil needs some refactoring, as it has become a large, hard-to-maintain, 
> hard-to-test class with a bunch of responsibilities:
>  * running qtests
>  * parsing/evaluating results
>  * masking/processing results
> First, I would extract a processing class from it (e.g. QOutProcessor or 
> something like that), which should be capable of being tested easily, and 
> contains all the logic what's needed to make the final q.out results from the 
> original q.out file. 
> This processing class shall remain untouched in case the testing 
> infrastructure (QTestUtil) is replaced or changed in some ways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18929) The method humanReadableInt in HiveStringUtils.java has a race condition.

2018-03-11 Thread Chaiyong Ragkhitwetsagul (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chaiyong Ragkhitwetsagul updated HIVE-18929:

Description: 
I found that the {{humanReadableInt(long number)}} method in the 
hive/common/src/java/org/apache/hive/common/util/HiveStringUtils.java file 
contains code which has a race condition as shown in Hadoop (issue tracking ID 
HADOOP-9252: https://issues.apache.org/jira/browse/HADOOP-9252). The fix can 
also be seen in the Hadoop code base.

I couldn't find a call to the method anywhere else in the code. But it might be 
worth to fix.

  was:
Hi. I found that the {{humanReadableInt(long number)}} method in the 
hive/common/src/java/org/apache/hive/common/util/HiveStringUtils.java file 
contains code which has a race condition as shown in Hadoop (issue tracking ID 
HADOOP-9252: https://issues.apache.org/jira/browse/HADOOP-9252). The fix can 
also be seen in the Hadoop code base.

I couldn't find a call to the method anywhere else in the code. But it might be 
worth to fix.


> The method humanReadableInt in HiveStringUtils.java has a race condition.
> -
>
> Key: HIVE-18929
> URL: https://issues.apache.org/jira/browse/HIVE-18929
> Project: Hive
>  Issue Type: Bug
>  Components: API
>Affects Versions: 2.3.2
>Reporter: Chaiyong Ragkhitwetsagul
>Priority: Minor
>
> I found that the {{humanReadableInt(long number)}} method in the 
> hive/common/src/java/org/apache/hive/common/util/HiveStringUtils.java file 
> contains code which has a race condition as shown in Hadoop (issue tracking 
> ID HADOOP-9252: https://issues.apache.org/jira/browse/HADOOP-9252). The fix 
> can also be seen in the Hadoop code base.
> I couldn't find a call to the method anywhere else in the code. But it might 
> be worth to fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18560) qtests: QTestUtil refactor/split - QOutProcessor

2018-03-11 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394514#comment-16394514
 ] 

Hive QA commented on HIVE-18560:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913937/HIVE-18560.02.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 13350 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursion.q,merge_constraint_notnull.q,load_part_nospec.q,clusterbyorderby.q,orc_type_promotion2.q,ctas_noperm_loc.q,udf_instr_wrong_args_len.q,invalid_create_tbl2.q,part_col_complex_type.q,authorization_drop_db_empty.q,smb_mapjoin_14.q,subquery_scalar_multi_rows.q,alter_partition_coltype_2columns.q,subquery_corr_in_agg.q,insert_overwrite_notnull_constraint.q,authorization_show_grant_otheruser_wtab.q,regex_col_groupby.q,udaf_collect_set_unsupported.q,ptf_negative_DuplicateWindowAlias.q,exim_22_export_authfail.q,udf_likeany_wrong1.q,groupby_key.q,ambiguous_col.q,groupby3_multi_distinct.q,authorization_alter_drop_ptn.q,invalid_cast_from_binary_5.q,show_create_table_does_not_exist.q,invalid_select_column.q,exim_20_managed_location_over_existing.q,interval_3.q,authorization_compile.q,join35.q,merge_negative_3.q,udf_concat_ws_wrong3.q,create_or_replace_view8.q,create_external_with_notnull_constraint.q,split_sample_out_of_range.q,materialized_view_no_transactional_rewrite.q,authorization_show_grant_otherrole.q,create_with_constraints_duplicate_name.q,invalid_stddev_samp_syntax.q,authorization_view_disable_cbo_7.q,autolocal1.q,avro_non_nullable_union.q,load_orc_negative_part.q,drop_view_failure1.q,columnstats_partlvl_invalid_values_autogather.q,exim_13_nonnative_import.q,alter_table_wrong_regex.q,add_partition_with_whitelist.q,udf_next_day_error_2.q,authorization_select.q,udf_trunc_error2.q,authorization_view_7.q,udf_format_number_wrong5.q,touch2.q,exim_03_nonpart_noncompat_colschema.q,orc_type_promotion1.q,lateral_view_alias.q,show_tables_bad_db1.q,unset_table_property.q,alter_non_native.q,nvl_mismatch_type.q,load_orc_negative3.q,authorization_create_role_no_admin.q,invalid_distinct1.q,authorization_grant_server.q,orc_type_promotion3_acid.q,show_tables_bad1.q,macro_unused_parameter.q,drop_invalid_constraint3.q,drop_partition_filter_failure.q,char_pad_convert_fail3.q,exim_23_import_exist_authfail.q,drop_invalid_constraint4.q,authorization_create_macro1.q,archive1.q,subquery_multiple_cols_in_select.q,change_hive_hdfs_session_path.q,udf_trunc_error3.q,invalid_variance_syntax.q,authorization_truncate_2.q,invalid_avg_syntax.q,invalid_select_column_with_tablename.q,mm_truncate_cols.q,groupby_grouping_sets1.q,druid_location.q,groupby2_multi_distinct.q,authorization_sba_drop_table.q,dynamic_partitions_with_whitelist.q,compare_string_bigint_2.q,udf_greatest_error_2.q,authorization_view_6.q,show_tablestatus.q,duplicate_alias_in_transform_schema.q,create_with_fk_uk_same_tab.q,udtf_not_supported3.q,alter_table_constraint_invalid_fk_col2.q,udtf_not_supported1.q,dbtxnmgr_notableunlock.q,ptf_negative_InvalidValueBoundary.q,alter_table_constraint_duplicate_pk.q,udf_printf_wrong4.q,create_view_failure9.q,udf_elt_wrong_type.q,selectDistinctStarNeg_1.q,invalid_mapjoin1.q,load_stored_as_dirs.q,input1.q,udf_sort_array_wrong1.q,invalid_distinct2.q,invalid_select_fn.q,authorization_role_grant_otherrole.q,archive4.q,load_nonpart_authfail.q,recursive_view.q,authorization_view_disable_cbo_1.q,desc_failure4.q,create_not_acid.q,udf_sort_array_wrong3.q,char_pad_convert_fail0.q,udf_map_values_arg_type.q,alter_view_failure6_2.q,alter_partition_change_col_nonexist.q,update_non_acid_table.q,authorization_view_disable_cbo_5.q,ct_noperm_loc.q,interval_1.q,authorization_show_grant_otheruser_all.q,authorization_view_2.q,show_tables_bad2.q,groupby_rollup2.q,truncate_column_seqfile.q,create_view_failure5.q,authorization_create_view.q,ptf_window_boundaries.q,ctasnullcol.q,input_part0_neg_2.q,create_or_replace_view1.q,udf_max.q,exim_01_nonpart_over_loaded.q,msck_repair_1.q,orc_change_fileformat_acid.q,udf_nonexistent_resource.q,exim_19_external_over_existing.q,serde_regex2.q,msck_repair_2.q,exim_06_nonpart_noncompat_storage.q,illegal_partition_type4.q,udf_sort_array_by_wrong1.q,create_or_replace_view5.q,windowing_leadlag_in_udaf.q,avro_decimal.q,materialized_view_upda

[jira] [Commented] (HIVE-18560) qtests: QTestUtil refactor/split - QOutProcessor

2018-03-11 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394495#comment-16394495
 ] 

Hive QA commented on HIVE-18560:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
11s{color} | {color:red} itests/util: The patch generated 6 new + 101 unchanged 
- 7 fixed = 107 total (was 108) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 52s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9594/dev-support/hive-personality.sh
 |
| git revision | master / dd0bc33 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9594/yetus/diff-checkstyle-itests_util.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9594/yetus/whitespace-eol.txt 
|
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9594/yetus/patch-asflicense-problems.txt
 |
| modules | C: itests/util ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9594/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> qtests: QTestUtil refactor/split - QOutProcessor
> 
>
> Key: HIVE-18560
> URL: https://issues.apache.org/jira/browse/HIVE-18560
> Project: Hive
>  Issue Type: Task
>  Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18560.01.patch, HIVE-18560.02.patch
>
>
> QTestUtil needs some refactoring, as it has become a large, hard-to-maintain, 
> hard-to-test class with a bunch of responsibilities:
>  * running qtests
>  * parsing/evaluating results
>  * masking/processing results
> First, I would extract a processing class from it (e.g. QOutProcessor or 
> something like that), which should be capable of being tested easily, and 
> contains all the logic what's needed to make the final q.out results from the 
> original q.out file. 
> This processing class shall remain untouched in case the testing 
> infrastructure (QTestUtil) is replaced or changed in some ways.



--
This message was sent by Atlassian JIRA
(

[jira] [Updated] (HIVE-18560) qtests: QTestUtil refactor/split - QOutProcessor

2018-03-11 Thread Laszlo Bodor (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-18560:

Attachment: HIVE-18560.02.patch

> qtests: QTestUtil refactor/split - QOutProcessor
> 
>
> Key: HIVE-18560
> URL: https://issues.apache.org/jira/browse/HIVE-18560
> Project: Hive
>  Issue Type: Task
>  Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18560.01.patch, HIVE-18560.02.patch
>
>
> QTestUtil needs some refactoring, as it has become a large, hard-to-maintain, 
> hard-to-test class with a bunch of responsibilities:
>  * running qtests
>  * parsing/evaluating results
>  * masking/processing results
> First, I would extract a processing class from it (e.g. QOutProcessor or 
> something like that), which should be capable of being tested easily, and 
> contains all the logic what's needed to make the final q.out results from the 
> original q.out file. 
> This processing class shall remain untouched in case the testing 
> infrastructure (QTestUtil) is replaced or changed in some ways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18862) qfiles: prepare .q files for using datasets

2018-03-11 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394477#comment-16394477
 ] 

Hive QA commented on HIVE-18862:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913926/HIVE-18862.02.patch

{color:green}SUCCESS:{color} +1 due to 1851 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 323 failed/errored test(s), 13324 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursion.q,merge_constraint_notnull.q,load_part_nospec.q,clusterbyorderby.q,orc_type_promotion2.q,ctas_noperm_loc.q,udf_instr_wrong_args_len.q,invalid_create_tbl2.q,part_col_complex_type.q,authorization_drop_db_empty.q,smb_mapjoin_14.q,subquery_scalar_multi_rows.q,alter_partition_coltype_2columns.q,subquery_corr_in_agg.q,insert_overwrite_notnull_constraint.q,authorization_show_grant_otheruser_wtab.q,regex_col_groupby.q,udaf_collect_set_unsupported.q,ptf_negative_DuplicateWindowAlias.q,exim_22_export_authfail.q,udf_likeany_wrong1.q,groupby_key.q,ambiguous_col.q,groupby3_multi_distinct.q,authorization_alter_drop_ptn.q,invalid_cast_from_binary_5.q,show_create_table_does_not_exist.q,invalid_select_column.q,exim_20_managed_location_over_existing.q,interval_3.q,authorization_compile.q,join35.q,merge_negative_3.q,udf_concat_ws_wrong3.q,create_or_replace_view8.q,create_external_with_notnull_constraint.q,split_sample_out_of_range.q,materialized_view_no_transactional_rewrite.q,authorization_show_grant_otherrole.q,create_with_constraints_duplicate_name.q,invalid_stddev_samp_syntax.q,authorization_view_disable_cbo_7.q,autolocal1.q,avro_non_nullable_union.q,load_orc_negative_part.q,drop_view_failure1.q,columnstats_partlvl_invalid_values_autogather.q,exim_13_nonnative_import.q,alter_table_wrong_regex.q,add_partition_with_whitelist.q,udf_next_day_error_2.q,authorization_select.q,udf_trunc_error2.q,authorization_view_7.q,udf_format_number_wrong5.q,touch2.q,exim_03_nonpart_noncompat_colschema.q,orc_type_promotion1.q,lateral_view_alias.q,show_tables_bad_db1.q,unset_table_property.q,alter_non_native.q,nvl_mismatch_type.q,load_orc_negative3.q,authorization_create_role_no_admin.q,invalid_distinct1.q,authorization_grant_server.q,orc_type_promotion3_acid.q,show_tables_bad1.q,macro_unused_parameter.q,drop_invalid_constraint3.q,drop_partition_filter_failure.q,char_pad_convert_fail3.q,exim_23_import_exist_authfail.q,drop_invalid_constraint4.q,authorization_create_macro1.q,archive1.q,subquery_multiple_cols_in_select.q,change_hive_hdfs_session_path.q,udf_trunc_error3.q,invalid_variance_syntax.q,authorization_truncate_2.q,invalid_avg_syntax.q,invalid_select_column_with_tablename.q,mm_truncate_cols.q,groupby_grouping_sets1.q,druid_location.q,groupby2_multi_distinct.q,authorization_sba_drop_table.q,dynamic_partitions_with_whitelist.q,compare_string_bigint_2.q,udf_greatest_error_2.q,authorization_view_6.q,show_tablestatus.q,duplicate_alias_in_transform_schema.q,create_with_fk_uk_same_tab.q,udtf_not_supported3.q,alter_table_constraint_invalid_fk_col2.q,udtf_not_supported1.q,dbtxnmgr_notableunlock.q,ptf_negative_InvalidValueBoundary.q,alter_table_constraint_duplicate_pk.q,udf_printf_wrong4.q,create_view_failure9.q,udf_elt_wrong_type.q,selectDistinctStarNeg_1.q,invalid_mapjoin1.q,load_stored_as_dirs.q,input1.q,udf_sort_array_wrong1.q,invalid_distinct2.q,invalid_select_fn.q,authorization_role_grant_otherrole.q,archive4.q,load_nonpart_authfail.q,recursive_view.q,authorization_view_disable_cbo_1.q,desc_failure4.q,create_not_acid.q,udf_sort_array_wrong3.q,char_pad_convert_fail0.q,udf_map_values_arg_type.q,alter_view_failure6_2.q,alter_partition_change_col_nonexist.q,update_non_acid_table.q,authorization_view_disable_cbo_5.q,ct_noperm_loc.q,interval_1.q,authorization_show_grant_otheruser_all.q,authorization_view_2.q,show_tables_bad2.q,groupby_rollup2.q,truncate_column_seqfile.q,create_view_failure5.q,authorization_create_view.q,ptf_window_boundaries.q,ctasnullcol.q,input_part0_neg_2.q,create_or_replace_view1.q,udf_max.q,exim_01_nonpart_over_loaded.q,msck_repair_1.q,orc_change_fileformat_acid.q,udf_nonexistent_resource.q,exim_19_external_over_existing.q,serde_regex2.q,msck_repair_2.q,exim_06_nonpart_noncompat_storage.q,illegal_partition_type4.q,udf_sort_array_by_wrong1.q,create_or_replace_view5.q,windowing_leadlag_in_udaf.q,avro_decimal.q,materialized_view_

[jira] [Commented] (HIVE-18862) qfiles: prepare .q files for using datasets

2018-03-11 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394472#comment-16394472
 ] 

Hive QA commented on HIVE-18862:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m  
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m  
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m  2s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9593/dev-support/hive-personality.sh
 |
| git revision | master / dd0bc33 |
| Default Java | 1.8.0_111 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9593/yetus/whitespace-eol.txt 
|
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9593/yetus/patch-asflicense-problems.txt
 |
| modules | C: . itests/util ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9593/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> qfiles: prepare .q files for using datasets
> ---
>
> Key: HIVE-18862
> URL: https://issues.apache.org/jira/browse/HIVE-18862
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch
>
>
> # Parse .q files for source table usage
>  # Add needed dataset annotations
>  # Remove setInitScript("q_test_init.sql"); from CliConfig



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18862) qfiles: prepare .q files for using datasets

2018-03-11 Thread Laszlo Bodor (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-18862:

Attachment: HIVE-18862.02.patch

> qfiles: prepare .q files for using datasets
> ---
>
> Key: HIVE-18862
> URL: https://issues.apache.org/jira/browse/HIVE-18862
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch
>
>
> # Parse .q files for source table usage
>  # Add needed dataset annotations
>  # Remove setInitScript("q_test_init.sql"); from CliConfig



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)