[jira] [Commented] (HIVE-13425) Fix partition addition in MSCK REPAIR TABLE command
[ https://issues.apache.org/jira/browse/HIVE-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295333#comment-15295333 ] Hive QA commented on HIVE-13425: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12805168/HIVE-13425.4.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 138 failed/errored test(s), 9256 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file TestMiniTezCliDriver-script_pipe.q-vector_decimal_aggregate.q-vector_data_types.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-tez_union_group_by.q-vector_auto_smb_mapjoin_14.q-union_fast_stats.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vector_interval_2.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-tez_fsstat.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vectorized_parquet.q-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more - did not produce a TEST-*.xml file TestNegativeCliDriver-udf_invalid.q-nopart_insert.q-insert_into_with_schema.q-and-734-more - did not produce a TEST-*.xml file TestSparkCliDriver-groupby6_map.q-join13.q-join_reorder3.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-ptf_rcfile.q-bucketmapjoin_negative.q-bucket_map_join_spark2.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-table_access_keys_stats.q-bucketsortoptimize_insert_4.q-runtime_skewjoin_mapjoin_spark.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_analyze_tbl_part org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_merge org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_into_with_schema org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lock3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lock4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_decode_name org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_special_char org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_quotedid_partition org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_merge2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_partitioned_date_time org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_in
[jira] [Commented] (HIVE-13425) Fix partition addition in MSCK REPAIR TABLE command
[ https://issues.apache.org/jira/browse/HIVE-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15297789#comment-15297789 ] Hive QA commented on HIVE-13425: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12805527/HIVE-13425.5.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 55 failed/errored test(s), 10043 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file TestMiniTezCliDriver-auto_sortmerge_join_7.q-orc_merge9.q-tez_union_dynamic_partition.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-cte_4.q-vector_non_string_partition.q-delete_where_non_partitioned.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-explainuser_4.q-update_after_multiple_inserts.q-mapreduce2.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vector_interval_2.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-tez_fsstat.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2 org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorParallelism org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking6 org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivilege
[jira] [Commented] (HIVE-13425) Fix partition addition in MSCK REPAIR TABLE command
[ https://issues.apache.org/jira/browse/HIVE-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298304#comment-15298304 ] Shinichi Yamashita commented on HIVE-13425: --- I confirmed the failed test by my environment. And I confirmed that all except for the following classes succeeds. {code} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore {code} I also carried out these tests by the state that my patch isn't applied. And I confirmed that all classes are failed. In other words, these failure isn't influence of my patch. > Fix partition addition in MSCK REPAIR TABLE command > --- > > Key: HIVE-13425 > URL: https://issues.apache.org/jira/browse/HIVE-13425 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.0.0 >Reporter: Shinichi Yamashita >Assignee: Shinichi Yamashita >Priority: Minor > Attachments: HIVE-13425.1.patch, HIVE-13425.2.patch, > HIVE-13425.3.patch, HIVE-13425.4.patch, HIVE-13425.5.patch > > > I create a table in following HiveQL. > {code} > hive> create table example (name string) partitioned by (id int); > {code} > And, I make some directories about this table in HDFS. > {code} > [root@hadoop ~]# hdfs dfs -ls -R /user/hive/warehouse/example > drwxr-xr-x - root hadoop 0 2016-04-05 22:21 > /user/hive/warehouse/example/id=1 > drwxr-xr-x - root hadoop 0 2016-04-05 22:22 > /user/hive/warehouse/example/id=1/id=2 > -rw-r--r-- 1 root hadoop 8 2016-04-05 22:22 > /user/hive/warehouse/example/id=1/id=2/example.txt > {code} > Next I executed MSCK REPAIR TABLE command and added a partition. And this > result became as follows. > {code} > [root@hadoop ~]# hive -e 'msck repair table example' > OK > Partitions not in metastore: example:id=1/id=2 > Repair: Added partition to metastore example:id=1/id=2 > Time taken: 1.243 seconds, Fetched: 2 row(s) > [root@hadoop ~]# hive -e 'show partitions example' > OK > id=2 > {code} > "id=1" should be a partition, but "id=2" was added. I will fix this problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13425) Fix partition addition in MSCK REPAIR TABLE command
[ https://issues.apache.org/jira/browse/HIVE-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369933#comment-15369933 ] Ashutosh Chauhan commented on HIVE-13425: - I don't understand following: {code} if (!path.toString().contains(checkPath)) { + String partKey = ""; + for (int i = 0; i < kvs.size(); i++) { +if (partKey.equals(kvs.get(i)[0])) { + partSpec.clear(); + break; +} +partKey = kvs.get(i)[0]; + } {code} your .4 patch made more sense to me. Can you describe motivation for diff between .4 & .5? > Fix partition addition in MSCK REPAIR TABLE command > --- > > Key: HIVE-13425 > URL: https://issues.apache.org/jira/browse/HIVE-13425 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.0.0 >Reporter: Shinichi Yamashita >Assignee: Shinichi Yamashita >Priority: Minor > Attachments: HIVE-13425.1.patch, HIVE-13425.2.patch, > HIVE-13425.3.patch, HIVE-13425.4.patch, HIVE-13425.5.patch > > > I create a table in following HiveQL. > {code} > hive> create table example (name string) partitioned by (id int); > {code} > And, I make some directories about this table in HDFS. > {code} > [root@hadoop ~]# hdfs dfs -ls -R /user/hive/warehouse/example > drwxr-xr-x - root hadoop 0 2016-04-05 22:21 > /user/hive/warehouse/example/id=1 > drwxr-xr-x - root hadoop 0 2016-04-05 22:22 > /user/hive/warehouse/example/id=1/id=2 > -rw-r--r-- 1 root hadoop 8 2016-04-05 22:22 > /user/hive/warehouse/example/id=1/id=2/example.txt > {code} > Next I executed MSCK REPAIR TABLE command and added a partition. And this > result became as follows. > {code} > [root@hadoop ~]# hive -e 'msck repair table example' > OK > Partitions not in metastore: example:id=1/id=2 > Repair: Added partition to metastore example:id=1/id=2 > Time taken: 1.243 seconds, Fetched: 2 row(s) > [root@hadoop ~]# hive -e 'show partitions example' > OK > id=2 > {code} > "id=1" should be a partition, but "id=2" was added. I will fix this problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13425) Fix partition addition in MSCK REPAIR TABLE command
[ https://issues.apache.org/jira/browse/HIVE-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15237167#comment-15237167 ] Hive QA commented on HIVE-13425: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12797912/HIVE-13425.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 214 failed/errored test(s), 9974 tests executed *Failed tests:* {noformat} TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_change_col org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_cascade org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_partition_drop org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_analyze_table_null_partition org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_partitioned org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_partitioned_native org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_schema_evolution_native org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog_dp org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cp_sel org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dbtxnmgr_query4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_all_partitioned org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_partitioned org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_whole_partition org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_partition_insert org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_partition_skip_default org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_merge org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_bucketing org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization_acid org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_full org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial_ndv org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_implicit_cast_during_insert org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert0 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_acid_dynamic_partition org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_into6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_into_with_schema org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_into_with_schema2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_dynamic_partitioned org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_acid org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_partitioned org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part9 org.apache.hadoop.hive.cli.T
[jira] [Commented] (HIVE-13425) Fix partition addition in MSCK REPAIR TABLE command
[ https://issues.apache.org/jira/browse/HIVE-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273878#comment-15273878 ] Hive QA commented on HIVE-13425: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12802105/HIVE-13425.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 9932 tests executed *Failed tests:* {noformat} TestCliDriver-gen_udf_example_add10.q-ppd_join4.q-union27.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-partition_timestamp.q-ppd_random.q-vector_outer_join5.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-ptf_general_queries.q-unionDistinct_1.q-groupby1_noskew.q-and-12-more - did not produce a TEST-*.xml file TestHWISessionManager - did not produce a TEST-*.xml file TestMiniLlapCliDriver - did not produce a TEST-*.xml file TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-insert_values_non_partitioned.q-join1.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-bucketsortoptimize_insert_7.q-smb_mapjoin_15.q-mapreduce1.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testNameMethods org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testNameMethods org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testNameMethods org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testNameMethods org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testNameMethods org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle {noformat} Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/193/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/193/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-193/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 20 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12802105 - PreCommit-HIVE-MASTER-Build > Fix partition addition in MSCK REPAIR TABLE command > --- > > Key: HIVE-13425 > URL: https://issues.apache.org/jira/browse/HIVE-13425 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.0.0 >Reporter: Shinichi Yamashita >Assignee: Shinichi Yamashita >Priority: Minor > Attachments: HIVE-13425.1.patch, HIVE-13425.2.patch > > > I create a table in following HiveQL. > {code} > hive> create table example (name string) partitioned by (id int); > {code} > And, I make some directories about this table in HDFS. > {code} > [root@hadoop ~]# hdfs dfs -ls -R /user/hive/warehouse/example > drwxr-xr-x - root hadoop 0 2016-04-05 22:21 > /user/hive/warehouse/example/id=1 > drwxr-xr-x - root hadoop 0 2016-04-05 22:22 > /user/hive/warehouse/example/id=1/id=2 > -rw-r--r-- 1 root hadoop 8 2016-04-05 22:22 > /user/hive/warehouse/example/id=1/id=2/example.txt > {code} > Next I executed MSCK REPAIR TABLE command and added a partition. And this > result became as follows. > {code} > [root@hadoop ~]# hive -e 'msck repair table example' > OK > Partitions not in metastore: example:id=1/id=2 > Repair: Added partition to metastore example:id=1/id=2 > Time taken: 1.243 seconds, Fetched: 2 row(s) > [root@hadoop ~]# hive -e 'show partitions example' > OK > id=2 > {code} > "id=1" should be a partition, but "id=2" was added. I will fix this problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13425) Fix partition addition in MSCK REPAIR TABLE command
[ https://issues.apache.org/jira/browse/HIVE-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275291#comment-15275291 ] Ashutosh Chauhan commented on HIVE-13425: - I am not sure if this is a right fix. e.g., if table is partitioned by (a int, b int) then for tbl/a=1/b=2 partspec will become (b=2/a=1) instead of (a=1/b=2) That would be incorrect. Also, can you take a look at HIVE-13703 where we might end up with dir structure after executing union query on Tez. I presume patch from here will also fix that. If so, can you add a test case for that too? > Fix partition addition in MSCK REPAIR TABLE command > --- > > Key: HIVE-13425 > URL: https://issues.apache.org/jira/browse/HIVE-13425 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.0.0 >Reporter: Shinichi Yamashita >Assignee: Shinichi Yamashita >Priority: Minor > Attachments: HIVE-13425.1.patch, HIVE-13425.2.patch, > HIVE-13425.3.patch > > > I create a table in following HiveQL. > {code} > hive> create table example (name string) partitioned by (id int); > {code} > And, I make some directories about this table in HDFS. > {code} > [root@hadoop ~]# hdfs dfs -ls -R /user/hive/warehouse/example > drwxr-xr-x - root hadoop 0 2016-04-05 22:21 > /user/hive/warehouse/example/id=1 > drwxr-xr-x - root hadoop 0 2016-04-05 22:22 > /user/hive/warehouse/example/id=1/id=2 > -rw-r--r-- 1 root hadoop 8 2016-04-05 22:22 > /user/hive/warehouse/example/id=1/id=2/example.txt > {code} > Next I executed MSCK REPAIR TABLE command and added a partition. And this > result became as follows. > {code} > [root@hadoop ~]# hive -e 'msck repair table example' > OK > Partitions not in metastore: example:id=1/id=2 > Repair: Added partition to metastore example:id=1/id=2 > Time taken: 1.243 seconds, Fetched: 2 row(s) > [root@hadoop ~]# hive -e 'show partitions example' > OK > id=2 > {code} > "id=1" should be a partition, but "id=2" was added. I will fix this problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13425) Fix partition addition in MSCK REPAIR TABLE command
[ https://issues.apache.org/jira/browse/HIVE-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15276950#comment-15276950 ] Hive QA commented on HIVE-13425: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12802825/HIVE-13425.3.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 9941 tests executed *Failed tests:* {noformat} TestCliDriver-gen_udf_example_add10.q-ppd_join4.q-union27.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-partition_timestamp.q-ppd_random.q-vector_outer_join5.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-ptf_general_queries.q-unionDistinct_1.q-groupby1_noskew.q-and-12-more - did not produce a TEST-*.xml file TestHWISessionManager - did not produce a TEST-*.xml file TestMiniLlapCliDriver - did not produce a TEST-*.xml file TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-join1.q-schema_evol_orc_nonvec_mapwork_part.q-mapjoin_decimal.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-bucketsortoptimize_insert_7.q-smb_mapjoin_15.q-mapreduce1.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle {noformat} Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/223/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/223/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-223/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 14 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12802825 - PreCommit-HIVE-MASTER-Build > Fix partition addition in MSCK REPAIR TABLE command > --- > > Key: HIVE-13425 > URL: https://issues.apache.org/jira/browse/HIVE-13425 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.0.0 >Reporter: Shinichi Yamashita >Assignee: Shinichi Yamashita >Priority: Minor > Attachments: HIVE-13425.1.patch, HIVE-13425.2.patch, > HIVE-13425.3.patch > > > I create a table in following HiveQL. > {code} > hive> create table example (name string) partitioned by (id int); > {code} > And, I make some directories about this table in HDFS. > {code} > [root@hadoop ~]# hdfs dfs -ls -R /user/hive/warehouse/example > drwxr-xr-x - root hadoop 0 2016-04-05 22:21 > /user/hive/warehouse/example/id=1 > drwxr-xr-x - root hadoop 0 2016-04-05 22:22 > /user/hive/warehouse/example/id=1/id=2 > -rw-r--r-- 1 root hadoop 8 2016-04-05 22:22 > /user/hive/warehouse/example/id=1/id=2/example.txt > {code} > Next I executed MSCK REPAIR TABLE command and added a partition. And this > result became as follows. > {code} > [root@hadoop ~]# hive -e 'msck repair table example' > OK > Partitions not in metastore: example:id=1/id=2 > Repair: Added partition to metastore example:id=1/id=2 > Time taken: 1.243 seconds, Fetched: 2 row(s) > [root@hadoop ~]# hive -e 'show partitions example' > OK > id=2 > {code} > "id=1" should be a partition, but "id=2" was added. I will fix this problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)