[jira] [Commented] (HIVE-15618) Change hive-blobstore tests to run with Tez by default
[
https://issues.apache.org/jira/browse/HIVE-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174369#comment-16174369
]
Hive QA commented on HIVE-15618:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12847373/HIVE-15618.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 11047 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_empty_into_blobstore]
(batchId=242)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
(batchId=242)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
(batchId=242)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
(batchId=242)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
(batchId=242)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
(batchId=242)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
(batchId=242)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=234)
org.apache.hadoop.hive.ql.parse.TestParseNegativeDriver.testCliDriver[wrong_distinct2]
(batchId=238)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6915/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6915/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6915/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12847373 - PreCommit-HIVE-Build
> Change hive-blobstore tests to run with Tez by default
> --
>
> Key: HIVE-15618
> URL: https://issues.apache.org/jira/browse/HIVE-15618
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Affects Versions: 2.2.0
>Reporter: Thomas Poepping
>Assignee: Thomas Poepping
> Attachments: HIVE-15618.patch
>
>
> Ever since the upgrade to Hive 2, Tez has been the default execution engine
> for Hive. To match that fact, it makes sense to run our tests against Tez,
> rather than MR. This should more fully validate functionality against what we
> consider to be Hive defaults.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174398#comment-16174398 ] Sergey Shelukhin commented on HIVE-15212: - Looks like multi insert overwrite is another gap. I actually noticed that the tests on the branch were left in commented out state, and only one section that was valid, has failed. Looks like it's not a stable failure either and depends on sequence of processing. I restored all the old queries and left it with incorrect results for now. Unfortunately because noone else is doing breaking work in branches, it's impossible to maintain the branch for a long time against continuous changes from replication, metastore stuff, etc. So, we are going to merge with this gap, and then fix it on master cc [~ekoifman] [~hagleitn] > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, > HIVE-15212.09.patch, HIVE-15212.10.patch, HIVE-15212.11.patch, > HIVE-15212.12.patch, HIVE-15212.12.patch, HIVE-15212.13.patch, > HIVE-15212.13.patch, HIVE-15212.14.patch, HIVE-15212.15.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17568) HiveJoinPushTransitivePredicatesRule may exchange predicates which are not valid on the other branch
[
https://issues.apache.org/jira/browse/HIVE-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-17568:
Attachment: HIVE-17568.02.patch
#2) add missing getType() to compare the basetype instead of the ref
> HiveJoinPushTransitivePredicatesRule may exchange predicates which are not
> valid on the other branch
>
>
> Key: HIVE-17568
> URL: https://issues.apache.org/jira/browse/HIVE-17568
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-17568.01.patch, HIVE-17568.02.patch
>
>
> Joining 2 tables on at least 1 column which is not the same type ;
> (integer/double for example).
> The calcite expressions require double/integer inputs which will became
> invalid if {{HiveJoinPushTransitivePredicatesRule}} pushes them to the other
> branch.
> query:
> {code}
> create table t1 (v string, k int);
> insert into t1 values ('people', 10), ('strangers', 20), ('parents', 30);
> create table t2 (v string, k double);
> insert into t2 values ('people', 10), ('strangers', 20), ('parents', 30);
> select * from t1 where t1.k in (select t2.k from t2 where t2.v='people') and
> t1.k<15;
> {code}
> results in:
> {code}
> java.lang.AssertionError: type mismatch:
> type1:
> DOUBLE
> type2:
> INTEGER
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:1841)
> at
> org.apache.calcite.rex.RexProgramBuilder$RegisterInputShuttle.visitInputRef(RexProgramBuilder.java:941)
> at
> org.apache.calcite.rex.RexProgramBuilder$RegisterInputShuttle.visitInputRef(RexProgramBuilder.java:919)
> at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
> at org.apache.calcite.rex.RexShuttle.visitList(RexShuttle.java:153)
> at org.apache.calcite.rex.RexShuttle.visitCall(RexShuttle.java:102)
> at
> org.apache.calcite.rex.RexProgramBuilder$RegisterShuttle.visitCall(RexProgramBuilder.java:884)
> at
> org.apache.calcite.rex.RexProgramBuilder$RegisterShuttle.visitCall(RexProgramBuilder.java:882)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:104)
> at
> org.apache.calcite.rex.RexProgramBuilder.registerInput(RexProgramBuilder.java:296)
> at
> org.apache.calcite.rex.RexProgramBuilder.addCondition(RexProgramBuilder.java:271)
> at
> org.apache.calcite.rel.rules.FilterMergeRule.createProgram(FilterMergeRule.java:98)
> at
> org.apache.calcite.rel.rules.FilterMergeRule.onMatch(FilterMergeRule.java:67)
> [...]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174403#comment-16174403 ] Sergey Shelukhin commented on HIVE-15212: - I cannot repro TestReplicationScenarios. These are new tests that might be unstable. mm_all failed due to the above gap. The rest of the failures are inherited from master Unfortunately there's another breaking change related to ACID... [~ekoifman] can you please forward port HIVE-15899 to the branch? I took a quick look but I'm out rest of the week and won't be able to do it today... you might be more familiar with its interplay with MM changes. After that we can probably do another run and merge... > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, > HIVE-15212.09.patch, HIVE-15212.10.patch, HIVE-15212.11.patch, > HIVE-15212.12.patch, HIVE-15212.12.patch, HIVE-15212.13.patch, > HIVE-15212.13.patch, HIVE-15212.14.patch, HIVE-15212.15.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17378) CBO: HiveReduceExpressionsWithStatsRule can operate on IS_NULL and IS_NOT_NULL
[
https://issues.apache.org/jira/browse/HIVE-17378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174438#comment-16174438
]
Zoltan Haindrich commented on HIVE-17378:
-
[~ashutoshc] Could you please take a look?
> CBO: HiveReduceExpressionsWithStatsRule can operate on IS_NULL and IS_NOT_NULL
> --
>
> Key: HIVE-17378
> URL: https://issues.apache.org/jira/browse/HIVE-17378
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Zoltan Haindrich
> Attachments: HIVE-17378.01.patch
>
>
> {code}
> * Currently we support the simplification of =, >=, <=, >, <, and
> * IN operations.
> */
> {code}
> IS_NULL and IS_NOT_NULL are closely related and can be processed by this rule.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17570) Fix view deletion related test failures (create_view.q etc)
[
https://issues.apache.org/jira/browse/HIVE-17570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174447#comment-16174447
]
Hive QA commented on HIVE-17570:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888128/HIVE-17570.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11047 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6916/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6916/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6916/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888128 - PreCommit-HIVE-Build
> Fix view deletion related test failures (create_view.q etc)
> ---
>
> Key: HIVE-17570
> URL: https://issues.apache.org/jira/browse/HIVE-17570
> Project: Hive
> Issue Type: Bug
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-17570.1.patch, HIVE-17570.2.patch
>
>
> Fixing the bug introduced by HIVE-17459. Sorry that did not capture that in a
> timely fashion.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17550) Remove unreferenced q.out-s
[ https://issues.apache.org/jira/browse/HIVE-17550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-17550: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) pushed to master. Thank you Ashutosh for the review! > Remove unreferenced q.out-s > --- > > Key: HIVE-17550 > URL: https://issues.apache.org/jira/browse/HIVE-17550 > Project: Hive > Issue Type: Improvement > Components: Tests >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Fix For: 3.0.0 > > Attachments: HIVE-17550.01.patch > > > it's sometimes a bit misleading to see q.out-s which are never even used.. > I'll also add a small utility which is able to remove them - and add a test > which will help to avoid them in the future -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17541) Move testing related methods from MetaStoreUtils to some testing related utility
[
https://issues.apache.org/jira/browse/HIVE-17541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-17541:
Fix Version/s: 3.0.0
> Move testing related methods from MetaStoreUtils to some testing related
> utility
>
>
> Key: HIVE-17541
> URL: https://issues.apache.org/jira/browse/HIVE-17541
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Fix For: 3.0.0
>
> Attachments: HIVE-17541.01.patch, HIVE-17541.02.patch
>
>
> MetaStoreUtils has a very wide range of methods...when the last time tried to
> do some modularization related with it - it always came back problematic :)
> The most usefull observation I made that it doesn't neccessarily needs the
> {{HMSHandler}} import.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17541) Move testing related methods from MetaStoreUtils to some testing related utility
[
https://issues.apache.org/jira/browse/HIVE-17541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-17541:
Resolution: Fixed
Status: Resolved (was: Patch Available)
pushed to master. Thank you Alan for the review!
> Move testing related methods from MetaStoreUtils to some testing related
> utility
>
>
> Key: HIVE-17541
> URL: https://issues.apache.org/jira/browse/HIVE-17541
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-17541.01.patch, HIVE-17541.02.patch
>
>
> MetaStoreUtils has a very wide range of methods...when the last time tried to
> do some modularization related with it - it always came back problematic :)
> The most usefull observation I made that it doesn't neccessarily needs the
> {{HMSHandler}} import.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17338) Utilities.get*Tasks multiple methods duplicate code
[ https://issues.apache.org/jira/browse/HIVE-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-17338: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) pushed to master. Thank yo [~ghajos] for picking this up! > Utilities.get*Tasks multiple methods duplicate code > --- > > Key: HIVE-17338 > URL: https://issues.apache.org/jira/browse/HIVE-17338 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Gergely Hajós > Fix For: 3.0.0 > > Attachments: HIVE-17338.1.patch, HIVE-17338.2.patch, > HIVE-17338.2.patch, HIVE-17338.3.patch > > > As discussed in https://github.com/apache/hive/pull/212/files, the 3 > functions can share a more general function. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17556) The test udf_mask_hash.q is failing
[
https://issues.apache.org/jira/browse/HIVE-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-17556:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
pushed to master. Thank you [~kuczoram] for fixing it!
> The test udf_mask_hash.q is failing
> ---
>
> Key: HIVE-17556
> URL: https://issues.apache.org/jira/browse/HIVE-17556
> Project: Hive
> Issue Type: Bug
> Components: Test
>Affects Versions: 3.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
> Fix For: 3.0.0
>
> Attachments: HIVE-17556.1.patch
>
>
> After committing
> [HIVE-17515|https://issues.apache.org/jira/browse/HIVE-17515], the
> udf_mask_hash.q test is failing with the following error message:
> {noformat}
> Error Message
> Client Execution succeeded but contained differences (error code = 1) after
> executing udf_mask_hash.q
> 34c34
> < expressions:
> '8b44d559dc5d60e4453c9b4edf2a455fbce054bb8504cd3eb9b5f391bd239c90' (type:
> string)
> ---
> > expressions: 'dd78d68ad1b23bde126812482dd70ac6' (type: string)
> 36c36
> < Statistics: Num rows: 1 Data size: 148 Basic stats: COMPLETE
> Column stats: COMPLETE
> ---
> > Statistics: Num rows: 1 Data size: 116 Basic stats: COMPLETE
> > Column stats: COMPLETE
> 61c61
> < 8b44d559dc5d60e4453c9b4edf2a455fbce054bb8504cd3eb9b5f391bd239c90
> 8b44d559dc5d60e4453c9b4edf2a455fbce054bb8504cd3eb9b5f391bd239c90
> 30a88603135d3a6f7a66b4f9193da1ab4423aed45fb8fe736c2f2a08977f2bdd
>
>
> NULL NULL NULL NULL NULL
> ---
> > dd78d68ad1b23bde126812482dd70ac6 dd78d68ad1b23bde126812482dd70ac6
> > 835735ba20f1297683efca69fabd0fba
> >
> >
> > NULL NULL NULL NULL NULL
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17472) Drop-partition for multi-level partition fails, if data does not exist.
[
https://issues.apache.org/jira/browse/HIVE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174522#comment-16174522
]
Hive QA commented on HIVE-17472:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888099/HIVE-17472.4-branch-2.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 58 failed/errored test(s), 9935 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=244)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=225)
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=167)
[acid_globallimit.q,alter_merge_2_orc.q]
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=173)
[infer_bucket_sort_reducers_power_two.q,list_bucket_dml_10.q,orc_merge9.q,orc_merge6.q,leftsemijoin_mr.q,bucket6.q,bucketmapjoin7.q,uber_reduce.q,empty_dir_in_table.q,vector_outer_join3.q,index_bitmap_auto.q,vector_outer_join2.q,vector_outer_join1.q,orc_merge1.q,orc_merge_diff_fs.q,load_hdfs_file_with_space_in_the_name.q,scriptfile1_win.q,quotedid_smb.q,truncate_column_buckets.q,orc_merge3.q]
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=174)
[infer_bucket_sort_num_buckets.q,gen_udf_example_add10.q,insert_overwrite_directory2.q,orc_merge5.q,bucketmapjoin6.q,import_exported_table.q,vector_outer_join0.q,orc_merge4.q,temp_table_external.q,orc_merge_incompat1.q,root_dir_external_table.q,constprog_semijoin.q,auto_sortmerge_join_16.q,schemeAuthority.q,index_bitmap3.q,external_table_with_space_in_location_path.q,parallel_orderby.q,infer_bucket_sort_map_operators.q,bucketizedhiveinputformat.q,remote_script.q]
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=175)
[scriptfile1.q,vector_outer_join5.q,file_with_header_footer.q,bucket4.q,input16_cc.q,bucket5.q,infer_bucket_sort_merge.q,constprog_partitioner.q,orc_merge2.q,reduce_deduplicate.q,schemeAuthority2.q,load_fs2.q,orc_merge8.q,orc_merge_incompat2.q,infer_bucket_sort_bucketed_table.q,vector_outer_join4.q,disable_merge_for_bucketing.q,vector_inner_join.q,orc_merge7.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=118)
[bucketmapjoin4.q,bucket_map_join_spark4.q,union21.q,groupby2_noskew.q,timestamp_2.q,date_join1.q,mergejoins.q,smb_mapjoin_11.q,auto_sortmerge_join_3.q,mapjoin_test_outer.q,vectorization_9.q,merge2.q,groupby6_noskew.q,auto_join_without_localtask.q,multi_join_union.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=119)
[join_cond_pushdown_unqual4.q,union_remove_7.q,join13.q,join_vc.q,groupby_cube1.q,bucket_map_join_spark2.q,sample3.q,smb_mapjoin_19.q,stats16.q,union23.q,union.q,union31.q,cbo_udf_udaf.q,ptf_decimal.q,bucketmapjoin2.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=120)
[parallel_join1.q,union27.q,union12.q,groupby7_map_multi_single_reducer.q,varchar_join1.q,join7.q,join_reorder4.q,skewjoinopt2.q,bucketsortoptimize_insert_2.q,smb_mapjoin_17.q,script_env_var1.q,groupby7_map.q,groupby3.q,bucketsortoptimize_insert_8.q,union20.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=121)
[ptf_general_queries.q,auto_join_reordering_values.q,sample2.q,join1.q,decimal_join.q,mapjoin_subquery2.q,join32_lessSize.q,mapjoin1.q,order2.q,skewjoinopt18.q,union_remove_18.q,join25.q,groupby9.q,bucketsortoptimize_insert_6.q,ctas.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=122)
[groupby_map_ppr.q,nullgroup4_multi_distinct.q,join_rc.q,union14.q,smb_mapjoin_12.q,vector_cast_constant.q,union_remove_4.q,auto_join11.q,load_dyn_part7.q,udaf_collect_set.q,vectorization_12.q,groupby_sort_skew_1.q,groupby_sort_skew_1_23.q,smb_mapjoin_25.q,skewjoinopt12.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=123)
[skewjoinopt15.q,auto_join18.q,list_bucket_dml_2.q,input1_limit.q,load_dyn_part3.q,union_remove_14.q,auto_sortmerge_join_14.q,auto_sortmerge_join_15.q,union10.q,bucket_map_join_tez2.q,groupby5_map_skew.q,join_reorder.q,sample1.q,bucketmapjoin8.q,union34.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=124)
[avro_joins.q,skewjoinopt16.q,auto_join14.q,vectorization_14.q,auto_join26.q,stats1.q,cbo_stats.q,auto_sortmerge_join_6.q,union22.q,union_remove_24.q,union_view.q,smb_mapjoin_22.q,stats15.q,ptf_matchpath.q,transform_ppr1.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=125)
[limit_pushdown2.q,skewjoin_noske
[jira] [Commented] (HIVE-17529) Bucket Map Join : Sets incorrect edge type causing execution failure
[
https://issues.apache.org/jira/browse/HIVE-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174598#comment-16174598
]
Hive QA commented on HIVE-17529:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888102/HIVE-17529.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11049 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6918/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6918/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6918/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888102 - PreCommit-HIVE-Build
> Bucket Map Join : Sets incorrect edge type causing execution failure
>
>
> Key: HIVE-17529
> URL: https://issues.apache.org/jira/browse/HIVE-17529
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-17529.1.patch, HIVE-17529.2.patch,
> HIVE-17529.3.patch, HIVE-17529.4.patch, HIVE-17529.5.patch
>
>
> If while traversing the tree to generate tasks, a bucket mapjoin may set its
> edge as CUSTOM_SIMPLE_EDGE against CUSTOM_EDGE if the bigtable is already not
> traversed causing Tez to assert and fail the vertex.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17533) Fill stats for temporary tables
[
https://issues.apache.org/jira/browse/HIVE-17533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-17533:
Summary: Fill stats for temporary tables (was: All stats are 0 for virtual
tables)
> Fill stats for temporary tables
> ---
>
> Key: HIVE-17533
> URL: https://issues.apache.org/jira/browse/HIVE-17533
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>
> Doing {{insert into t values (...)}} initializes a temporary table with 0
> stats.
> This could be made accurate since the contents of the table is already known.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17527) Support replication for rename/move table across database
[
https://issues.apache.org/jira/browse/HIVE-17527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174691#comment-16174691
]
Hive QA commented on HIVE-17527:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888083/HIVE-17527.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11051 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6919/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6919/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6919/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888083 - PreCommit-HIVE-Build
> Support replication for rename/move table across database
> -
>
> Key: HIVE-17527
> URL: https://issues.apache.org/jira/browse/HIVE-17527
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Labels: DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17527.01.patch, HIVE-17527.01.patch
>
>
> Rename/move table across database should be supported for replication. The
> scenario is as follows.
> 1. Create 2 databases (db1 and db2) in source cluster.
> 2. Create the table db1.tbl1.
> 3. Run bootstrap replication for db1 and db2 to target cluster.
> 4. Rename db1.tbl1 to db2.tbl1 in source.
> 5. Run incremental replication for both db1 and db2.
> - db1 dump missed the rename table operation as no event is generated for
> db1. So, table exist after load.
> - db2 load skips the rename event as the source table is missing in target.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated HIVE-17139: -- Attachment: HIVE-17139.12.patch > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.10.patch, HIVE-17139.11.patch, > HIVE-17139.12.patch, HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch, > HIVE-17139.6.patch, HIVE-17139.7.patch, HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17527) Support replication for rename/move table across database
[
https://issues.apache.org/jira/browse/HIVE-17527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174831#comment-16174831
]
Hive QA commented on HIVE-17527:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888083/HIVE-17527.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11051 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6920/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6920/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6920/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888083 - PreCommit-HIVE-Build
> Support replication for rename/move table across database
> -
>
> Key: HIVE-17527
> URL: https://issues.apache.org/jira/browse/HIVE-17527
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Labels: DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17527.01.patch, HIVE-17527.01.patch
>
>
> Rename/move table across database should be supported for replication. The
> scenario is as follows.
> 1. Create 2 databases (db1 and db2) in source cluster.
> 2. Create the table db1.tbl1.
> 3. Run bootstrap replication for db1 and db2 to target cluster.
> 4. Rename db1.tbl1 to db2.tbl1 in source.
> 5. Run incremental replication for both db1 and db2.
> - db1 dump missed the rename table operation as no event is generated for
> db1. So, table exist after load.
> - db2 load skips the rename event as the source table is missing in target.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17529) Bucket Map Join : Sets incorrect edge type causing execution failure
[ https://issues.apache.org/jira/browse/HIVE-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174916#comment-16174916 ] Deepak Jaiswal commented on HIVE-17529: --- The updated query worked. The test is now passing. > Bucket Map Join : Sets incorrect edge type causing execution failure > > > Key: HIVE-17529 > URL: https://issues.apache.org/jira/browse/HIVE-17529 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-17529.1.patch, HIVE-17529.2.patch, > HIVE-17529.3.patch, HIVE-17529.4.patch, HIVE-17529.5.patch > > > If while traversing the tree to generate tasks, a bucket mapjoin may set its > edge as CUSTOM_SIMPLE_EDGE against CUSTOM_EDGE if the bigtable is already not > traversed causing Tez to assert and fail the vertex. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17529) Bucket Map Join : Sets incorrect edge type causing execution failure
[ https://issues.apache.org/jira/browse/HIVE-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174916#comment-16174916 ] Deepak Jaiswal edited comment on HIVE-17529 at 9/21/17 3:34 PM: The updated query worked. The test is now passing. [~jdere] can you please review? was (Author: djaiswal): The updated query worked. The test is now passing. > Bucket Map Join : Sets incorrect edge type causing execution failure > > > Key: HIVE-17529 > URL: https://issues.apache.org/jira/browse/HIVE-17529 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-17529.1.patch, HIVE-17529.2.patch, > HIVE-17529.3.patch, HIVE-17529.4.patch, HIVE-17529.5.patch > > > If while traversing the tree to generate tasks, a bucket mapjoin may set its > edge as CUSTOM_SIMPLE_EDGE against CUSTOM_EDGE if the bigtable is already not > traversed causing Tez to assert and fail the vertex. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17110) BucketCodec should enforce value ranges
[
https://issues.apache.org/jira/browse/HIVE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174962#comment-16174962
]
Hive QA commented on HIVE-17110:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888160/HIVE-17110.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11049 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6921/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6921/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6921/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888160 - PreCommit-HIVE-Build
> BucketCodec should enforce value ranges
> ---
>
> Key: HIVE-17110
> URL: https://issues.apache.org/jira/browse/HIVE-17110
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 3.0.0
>
> Attachments: HIVE-17110.01.patch, HIVE-17110.02.patch
>
>
> BucketCodec added in HIVE-16832 can only handle statement id/bucket id values
> in a particular range. Add an assert to check that values are indeed in that
> range.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16084) SHOW COMPACTIONS should display CompactionID
[ https://issues.apache.org/jira/browse/HIVE-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174967#comment-16174967 ] Eugene Koifman commented on HIVE-16084: --- all failures age > 1 > SHOW COMPACTIONS should display CompactionID > > > Key: HIVE-16084 > URL: https://issues.apache.org/jira/browse/HIVE-16084 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Minor > Attachments: HIVE-16084.01.patch, HIVE-16084.02.patch > > > together with HIVE-13353 it will let users search for specific job -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17110) BucketCodec should enforce value ranges
[ https://issues.apache.org/jira/browse/HIVE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174975#comment-16174975 ] Eugene Koifman commented on HIVE-17110: --- no related failures > BucketCodec should enforce value ranges > --- > > Key: HIVE-17110 > URL: https://issues.apache.org/jira/browse/HIVE-17110 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 3.0.0 > > Attachments: HIVE-17110.01.patch, HIVE-17110.02.patch > > > BucketCodec added in HIVE-16832 can only handle statement id/bucket id values > in a particular range. Add an assert to check that values are indeed in that > range. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174977#comment-16174977 ] Eugene Koifman commented on HIVE-15212: --- [~wei.zheng] told me that IOW on MM branch wasn't complete since it uses base_N/ concept which MM doesn't yet understand. > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, > HIVE-15212.09.patch, HIVE-15212.10.patch, HIVE-15212.11.patch, > HIVE-15212.12.patch, HIVE-15212.12.patch, HIVE-15212.13.patch, > HIVE-15212.13.patch, HIVE-15212.14.patch, HIVE-15212.15.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17371) Move tokenstores to metastore module
[
https://issues.apache.org/jira/browse/HIVE-17371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17371:
---
Summary: Move tokenstores to metastore module (was: Fix DBTokenStore and
ZKTokenStore for the stand-alone metastore)
> Move tokenstores to metastore module
>
>
> Key: HIVE-17371
> URL: https://issues.apache.org/jira/browse/HIVE-17371
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> The {{getTokenStore}} method will not work for the {{DBTokenStore}} and
> {{ZKTokenStore}} since they implement
> {{org.apache.hadoop.hive.thrift.DelegationTokenStore}} instead of
> {{org.apache.hadoop.hive.metastore.security.DelegationTokenStore}}
> {code}
> private DelegationTokenStore getTokenStore(Configuration conf) throws
> IOException {
> String tokenStoreClassName =
> MetastoreConf.getVar(conf,
> MetastoreConf.ConfVars.DELEGATION_TOKEN_STORE_CLS, "");
> // The second half of this if is to catch cases where users are passing
> in a HiveConf for
> // configuration. It will have set the default value of
> // "hive.cluster.delegation.token.store .class" to
> // "org.apache.hadoop.hive.thrift.MemoryTokenStore" as part of its
> construction. But this is
> // the hive-shims version of the memory store. We want to convert this
> to our default value.
> if (StringUtils.isBlank(tokenStoreClassName) ||
>
> "org.apache.hadoop.hive.thrift.MemoryTokenStore".equals(tokenStoreClassName))
> {
> return new MemoryTokenStore();
> }
> try {
> Class storeClass =
>
> Class.forName(tokenStoreClassName).asSubclass(DelegationTokenStore.class);
> return ReflectionUtils.newInstance(storeClass, conf);
> } catch (ClassNotFoundException e) {
> throw new IOException("Error initializing delegation token store: " +
> tokenStoreClassName, e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17570) Fix view deletion related test failures (create_view.q etc)
[ https://issues.apache.org/jira/browse/HIVE-17570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175004#comment-16175004 ] Tao Li commented on HIVE-17570: --- Test result looks good. > Fix view deletion related test failures (create_view.q etc) > --- > > Key: HIVE-17570 > URL: https://issues.apache.org/jira/browse/HIVE-17570 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17570.1.patch, HIVE-17570.2.patch > > > Fixing the bug introduced by HIVE-17459. Sorry that did not capture that in a > timely fashion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17371) Move tokenstores to metastore module
[
https://issues.apache.org/jira/browse/HIVE-17371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17371:
---
Attachment: HIVE-17371.01.patch
Attaching the first version of the patch to trigger full suite of tests.
I think it makes more sense to move the tokenstore implementations to metastore
instead of having two copies of each in Hive and metastore. The TokenManager
and SecretManager classes can be kept as is to each consuming applications like
HS2 or Metastore itself so that future modifications can be done independently.
Here is the approach I followed.
1. Moved the {{HiveDelegationTokenManager}} and
{{TokenStoreDelegationTokenSecretManager}} from {{shims-common}} to {{service}}
module since Metastore anyways has its own copy of TokenManager and Metastore
cannot have a dependency on shims-common.
2. Moved the {{MemoryTokenStore}}, {{ZooKeeperTokenStore}} and {{DBTokenStore}}
to standalone-metastore module. In order for these tokenstores to be consumable
for multiple applications like HS2 and Metastore (and hopefully other
applications in the future) we need to operate of higher level abstraction of
{{AbstractDelegationTokenIdentifier}} and {{DelegationTokenInformation}}.
3. Each application can define its DelegationTokenIndentifier like Hive already
does in {{DelegationTokenIdentifier}} (may be we should rename this to
{{HiveDelegationTokenIdentifier}} to make it more explicit. Similarly,
metastore has its own implementation of DelegationTokenIdentifier in
{{DelegationTokenIdentifier}} (again we should rename it possibly to say
MetastoreDelegationTokenIndentifier)
4. Modified the shims-common classes of {{DelegationTokenSecretManager}} and
{{DelegationTokenStore}} to operate on hadoop level abstractions instead of
Hive defined ones.
> Move tokenstores to metastore module
>
>
> Key: HIVE-17371
> URL: https://issues.apache.org/jira/browse/HIVE-17371
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17371.01.patch
>
>
> The {{getTokenStore}} method will not work for the {{DBTokenStore}} and
> {{ZKTokenStore}} since they implement
> {{org.apache.hadoop.hive.thrift.DelegationTokenStore}} instead of
> {{org.apache.hadoop.hive.metastore.security.DelegationTokenStore}}
> {code}
> private DelegationTokenStore getTokenStore(Configuration conf) throws
> IOException {
> String tokenStoreClassName =
> MetastoreConf.getVar(conf,
> MetastoreConf.ConfVars.DELEGATION_TOKEN_STORE_CLS, "");
> // The second half of this if is to catch cases where users are passing
> in a HiveConf for
> // configuration. It will have set the default value of
> // "hive.cluster.delegation.token.store .class" to
> // "org.apache.hadoop.hive.thrift.MemoryTokenStore" as part of its
> construction. But this is
> // the hive-shims version of the memory store. We want to convert this
> to our default value.
> if (StringUtils.isBlank(tokenStoreClassName) ||
>
> "org.apache.hadoop.hive.thrift.MemoryTokenStore".equals(tokenStoreClassName))
> {
> return new MemoryTokenStore();
> }
> try {
> Class storeClass =
>
> Class.forName(tokenStoreClassName).asSubclass(DelegationTokenStore.class);
> return ReflectionUtils.newInstance(storeClass, conf);
> } catch (ClassNotFoundException e) {
> throw new IOException("Error initializing delegation token store: " +
> tokenStoreClassName, e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17371) Move tokenstores to metastore module
[
https://issues.apache.org/jira/browse/HIVE-17371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17371:
---
Status: Patch Available (was: Open)
> Move tokenstores to metastore module
>
>
> Key: HIVE-17371
> URL: https://issues.apache.org/jira/browse/HIVE-17371
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17371.01.patch
>
>
> The {{getTokenStore}} method will not work for the {{DBTokenStore}} and
> {{ZKTokenStore}} since they implement
> {{org.apache.hadoop.hive.thrift.DelegationTokenStore}} instead of
> {{org.apache.hadoop.hive.metastore.security.DelegationTokenStore}}
> {code}
> private DelegationTokenStore getTokenStore(Configuration conf) throws
> IOException {
> String tokenStoreClassName =
> MetastoreConf.getVar(conf,
> MetastoreConf.ConfVars.DELEGATION_TOKEN_STORE_CLS, "");
> // The second half of this if is to catch cases where users are passing
> in a HiveConf for
> // configuration. It will have set the default value of
> // "hive.cluster.delegation.token.store .class" to
> // "org.apache.hadoop.hive.thrift.MemoryTokenStore" as part of its
> construction. But this is
> // the hive-shims version of the memory store. We want to convert this
> to our default value.
> if (StringUtils.isBlank(tokenStoreClassName) ||
>
> "org.apache.hadoop.hive.thrift.MemoryTokenStore".equals(tokenStoreClassName))
> {
> return new MemoryTokenStore();
> }
> try {
> Class storeClass =
>
> Class.forName(tokenStoreClassName).asSubclass(DelegationTokenStore.class);
> return ReflectionUtils.newInstance(storeClass, conf);
> } catch (ClassNotFoundException e) {
> throw new IOException("Error initializing delegation token store: " +
> tokenStoreClassName, e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17562) ACID 1.0 + ETL strategy should treat empty compacted files as uncovered deltas
[
https://issues.apache.org/jira/browse/HIVE-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175059#comment-16175059
]
Hive QA commented on HIVE-17562:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888127/HIVE-17562-branch-2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10617 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=153)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[merge_negative_5]
(batchId=88)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf]
(batchId=125)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=176)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6922/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6922/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6922/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888127 - PreCommit-HIVE-Build
> ACID 1.0 + ETL strategy should treat empty compacted files as uncovered deltas
> --
>
> Key: HIVE-17562
> URL: https://issues.apache.org/jira/browse/HIVE-17562
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.4.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17562.1.branch-2.patch, HIVE-17562-branch-2.patch
>
>
> In branch-2, with ACID 1.0, following sequence will result in incorrect
> results
> 1) Set split strategy to ETL
> 2) Insert some rows
> 3) Delete all rows
> 4) Alter table compact MAJOR
> 5) Insert some rows
> 6) Select * query will not return any rows that is written at last (step 5)
> The reason for that, compaction essentially voids the first insert in step 2.
> Now when ETL split strategy is chosen, there will not be any stripes in the
> base files. So no split gets generated and any subsequent deltas gets ignored.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16898) Validation of source file after distcp in repl load
[ https://issues.apache.org/jira/browse/HIVE-16898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-16898: -- Attachment: HIVE-16898.7.patch > Validation of source file after distcp in repl load > > > Key: HIVE-16898 > URL: https://issues.apache.org/jira/browse/HIVE-16898 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Daniel Dai > Fix For: 3.0.0 > > Attachments: HIVE-16898.1.patch, HIVE-16898.2.patch, > HIVE-16898.3.patch, HIVE-16898.4.patch, HIVE-16898.5.patch, > HIVE-16898.6.patch, HIVE-16898.7.patch > > > time between deciding the source and destination path for distcp to invoking > of distcp can have a change of the source file, hence distcp might copy the > wrong file to destination, hence we should an additional check on the > checksum of the source file path after distcp finishes to make sure the path > didnot change during the copy process. if it has take additional steps to > delete the previous file on destination and copy the new source and repeat > the same process as above till we copy the correct file. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17574) Avoid multiple copies of HDFS-based jars when localizing job-jars
[
https://issues.apache.org/jira/browse/HIVE-17574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan reassigned HIVE-17574:
---
> Avoid multiple copies of HDFS-based jars when localizing job-jars
> -
>
> Key: HIVE-17574
> URL: https://issues.apache.org/jira/browse/HIVE-17574
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0, 3.0.0, 2.4.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>
> Raising this on behalf of [~selinazh]. (For my own reference: YHIVE-1035.)
> This has to do with the classpaths of Hive actions run from Oozie, and
> affects scripts that adds jars/resources from HDFS locations.
> As part of Oozie's "sharelib" deploys, foundation jars (such as Hive jars)
> tend to be stored in HDFS paths, as are any custom user-libraries used in
> workflows. An {{ADD JAR|FILE|ARCHIVE}} statement in a Hive script causes the
> following steps to occur:
> # Files are downloaded from HDFS to local temp dir.
> # UDFs are resolved/validated.
> # All jars/files, including those just downloaded from HDFS, are shipped
> right back to HDFS-based scratch-directories, for job submission.
> This is wasteful and time-consuming. #3 above should skip shipping HDFS-based
> resources, and add those directly to the Tez session.
> We have a patch that's being used internally at Yahoo.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17472) Drop-partition for multi-level partition fails, if data does not exist.
[
https://issues.apache.org/jira/browse/HIVE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175183#comment-16175183
]
Mithun Radhakrishnan commented on HIVE-17472:
-
Ok, looks like the dummy patches are producing the same failures on
{{branch-2.2}} as the fix. The failing tests seem to run locally, save for the
{{TestSparkCliDriver}}.
I'm +1. Just running a couple of double-checks before I check this in.
> Drop-partition for multi-level partition fails, if data does not exist.
> ---
>
> Key: HIVE-17472
> URL: https://issues.apache.org/jira/browse/HIVE-17472
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17472.1.patch, HIVE-17472.2-branch-2.patch,
> HIVE-17472.2.patch, HIVE-17472.3-branch-2.2.patch,
> HIVE-17472.3-branch-2.patch, HIVE-17472.3.patch,
> HIVE-17472.4-branch-2.2.patch, HIVE-17472.4.patch
>
>
> Raising this on behalf of [~cdrome] and [~selinazh].
> Here's how to reproduce the problem:
> {code:sql}
> CREATE TABLE foobar ( foo STRING, bar STRING ) PARTITIONED BY ( dt STRING,
> region STRING ) STORED AS RCFILE LOCATION '/tmp/foobar';
> ALTER TABLE foobar ADD PARTITION ( dt='1', region='A' ) ;
> dfs -rm -R -skipTrash /tmp/foobar/dt=1;
> ALTER TABLE foobar DROP PARTITION ( dt='1' );
> {code}
> This causes a client-side error as follows:
> {code}
> 15/02/26 23:08:32 ERROR exec.DDLTask:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unknown error. Please check
> logs.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17472) Drop-partition for multi-level partition fails, if data does not exist.
[
https://issues.apache.org/jira/browse/HIVE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175183#comment-16175183
]
Mithun Radhakrishnan edited comment on HIVE-17472 at 9/21/17 5:55 PM:
--
Ok, looks like the dummy patches are producing the same failures on
{{branch-2.2}} as with the fix. The failing tests seem to pass locally (or fail
in the same manner as without the patch), save for the {{TestSparkCliDriver}}.
I'm +1. Just running a couple of double-checks before I check this in.
was (Author: mithun):
Ok, looks like the dummy patches are producing the same failures on
{{branch-2.2}} as the fix. The failing tests seem to run locally, save for the
{{TestSparkCliDriver}}.
I'm +1. Just running a couple of double-checks before I check this in.
> Drop-partition for multi-level partition fails, if data does not exist.
> ---
>
> Key: HIVE-17472
> URL: https://issues.apache.org/jira/browse/HIVE-17472
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17472.1.patch, HIVE-17472.2-branch-2.patch,
> HIVE-17472.2.patch, HIVE-17472.3-branch-2.2.patch,
> HIVE-17472.3-branch-2.patch, HIVE-17472.3.patch,
> HIVE-17472.4-branch-2.2.patch, HIVE-17472.4.patch
>
>
> Raising this on behalf of [~cdrome] and [~selinazh].
> Here's how to reproduce the problem:
> {code:sql}
> CREATE TABLE foobar ( foo STRING, bar STRING ) PARTITIONED BY ( dt STRING,
> region STRING ) STORED AS RCFILE LOCATION '/tmp/foobar';
> ALTER TABLE foobar ADD PARTITION ( dt='1', region='A' ) ;
> dfs -rm -R -skipTrash /tmp/foobar/dt=1;
> ALTER TABLE foobar DROP PARTITION ( dt='1' );
> {code}
> This causes a client-side error as follows:
> {code}
> 15/02/26 23:08:32 ERROR exec.DDLTask:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unknown error. Please check
> logs.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[
https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175199#comment-16175199
]
Hive QA commented on HIVE-17558:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888126/HIVE-17558.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11049 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=291)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6923/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6923/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6923/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888126 - PreCommit-HIVE-Build
> Skip non-native/temporary tables for constraint related scenarios
> -
>
> Key: HIVE-17558
> URL: https://issues.apache.org/jira/browse/HIVE-17558
> Project: Hive
> Issue Type: Bug
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-17558.1.patch, HIVE-17558.2.patch
>
>
> The change would be similar to HIVE-17422.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17110) BucketCodec should enforce value ranges
[ https://issues.apache.org/jira/browse/HIVE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175206#comment-16175206 ] Gopal V commented on HIVE-17110: LGTM - +1 > BucketCodec should enforce value ranges > --- > > Key: HIVE-17110 > URL: https://issues.apache.org/jira/browse/HIVE-17110 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 3.0.0 > > Attachments: HIVE-17110.01.patch, HIVE-17110.02.patch > > > BucketCodec added in HIVE-16832 can only handle statement id/bucket id values > in a particular range. Add an assert to check that values are indeed in that > range. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17529) Bucket Map Join : Sets incorrect edge type causing execution failure
[ https://issues.apache.org/jira/browse/HIVE-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175211#comment-16175211 ] Jason Dere commented on HIVE-17529: --- +1 > Bucket Map Join : Sets incorrect edge type causing execution failure > > > Key: HIVE-17529 > URL: https://issues.apache.org/jira/browse/HIVE-17529 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-17529.1.patch, HIVE-17529.2.patch, > HIVE-17529.3.patch, HIVE-17529.4.patch, HIVE-17529.5.patch > > > If while traversing the tree to generate tasks, a bucket mapjoin may set its > edge as CUSTOM_SIMPLE_EDGE against CUSTOM_EDGE if the bigtable is already not > traversed causing Tez to assert and fail the vertex. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16298) Add config to specify multi-column joins have correlated columns
[ https://issues.apache.org/jira/browse/HIVE-16298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-16298: -- Fix Version/s: (was: 2.3.0) 3.0.0 > Add config to specify multi-column joins have correlated columns > > > Key: HIVE-16298 > URL: https://issues.apache.org/jira/browse/HIVE-16298 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Siddharth Seth > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-16298.1.patch, HIVE-16298.2.patch > > > The default row estimation for multi-key joins divides the row estimate by > the product of the NDVs for each join column, which can cause the row > estimate to be low. Try adding a config to assume the columns are correlated, > where we only divide the row estimate by the largest NDV. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175224#comment-16175224 ] Tao Li commented on HIVE-17558: --- Test result looks good. > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch, HIVE-17558.2.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17472) Drop-partition for multi-level partition fails, if data does not exist.
[
https://issues.apache.org/jira/browse/HIVE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17472:
Attachment: (was: HIVE-17472.4.patch)
> Drop-partition for multi-level partition fails, if data does not exist.
> ---
>
> Key: HIVE-17472
> URL: https://issues.apache.org/jira/browse/HIVE-17472
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17472.1.patch, HIVE-17472.2-branch-2.patch,
> HIVE-17472.2.patch, HIVE-17472.3-branch-2.2.patch,
> HIVE-17472.3-branch-2.patch, HIVE-17472.3.patch
>
>
> Raising this on behalf of [~cdrome] and [~selinazh].
> Here's how to reproduce the problem:
> {code:sql}
> CREATE TABLE foobar ( foo STRING, bar STRING ) PARTITIONED BY ( dt STRING,
> region STRING ) STORED AS RCFILE LOCATION '/tmp/foobar';
> ALTER TABLE foobar ADD PARTITION ( dt='1', region='A' ) ;
> dfs -rm -R -skipTrash /tmp/foobar/dt=1;
> ALTER TABLE foobar DROP PARTITION ( dt='1' );
> {code}
> This causes a client-side error as follows:
> {code}
> 15/02/26 23:08:32 ERROR exec.DDLTask:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unknown error. Please check
> logs.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17472) Drop-partition for multi-level partition fails, if data does not exist.
[
https://issues.apache.org/jira/browse/HIVE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175223#comment-16175223
]
Mithun Radhakrishnan commented on HIVE-17472:
-
Committed to {{master}}, {{branch-2}}, and {{branch-2.2}}. Thank you for this
fix, [~selinazh], and [~cdrome]!
> Drop-partition for multi-level partition fails, if data does not exist.
> ---
>
> Key: HIVE-17472
> URL: https://issues.apache.org/jira/browse/HIVE-17472
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17472.1.patch, HIVE-17472.2-branch-2.patch,
> HIVE-17472.2.patch, HIVE-17472.3-branch-2.2.patch,
> HIVE-17472.3-branch-2.patch, HIVE-17472.3.patch
>
>
> Raising this on behalf of [~cdrome] and [~selinazh].
> Here's how to reproduce the problem:
> {code:sql}
> CREATE TABLE foobar ( foo STRING, bar STRING ) PARTITIONED BY ( dt STRING,
> region STRING ) STORED AS RCFILE LOCATION '/tmp/foobar';
> ALTER TABLE foobar ADD PARTITION ( dt='1', region='A' ) ;
> dfs -rm -R -skipTrash /tmp/foobar/dt=1;
> ALTER TABLE foobar DROP PARTITION ( dt='1' );
> {code}
> This causes a client-side error as follows:
> {code}
> 15/02/26 23:08:32 ERROR exec.DDLTask:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unknown error. Please check
> logs.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17472) Drop-partition for multi-level partition fails, if data does not exist.
[
https://issues.apache.org/jira/browse/HIVE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17472:
Attachment: (was: HIVE-17472.4-branch-2.2.patch)
> Drop-partition for multi-level partition fails, if data does not exist.
> ---
>
> Key: HIVE-17472
> URL: https://issues.apache.org/jira/browse/HIVE-17472
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17472.1.patch, HIVE-17472.2-branch-2.patch,
> HIVE-17472.2.patch, HIVE-17472.3-branch-2.2.patch,
> HIVE-17472.3-branch-2.patch, HIVE-17472.3.patch
>
>
> Raising this on behalf of [~cdrome] and [~selinazh].
> Here's how to reproduce the problem:
> {code:sql}
> CREATE TABLE foobar ( foo STRING, bar STRING ) PARTITIONED BY ( dt STRING,
> region STRING ) STORED AS RCFILE LOCATION '/tmp/foobar';
> ALTER TABLE foobar ADD PARTITION ( dt='1', region='A' ) ;
> dfs -rm -R -skipTrash /tmp/foobar/dt=1;
> ALTER TABLE foobar DROP PARTITION ( dt='1' );
> {code}
> This causes a client-side error as follows:
> {code}
> 15/02/26 23:08:32 ERROR exec.DDLTask:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unknown error. Please check
> logs.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17472) Drop-partition for multi-level partition fails, if data does not exist.
[
https://issues.apache.org/jira/browse/HIVE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17472:
Resolution: Fixed
Fix Version/s: 2.2.1
2.4.0
3.0.0
Status: Resolved (was: Patch Available)
> Drop-partition for multi-level partition fails, if data does not exist.
> ---
>
> Key: HIVE-17472
> URL: https://issues.apache.org/jira/browse/HIVE-17472
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-17472.1.patch, HIVE-17472.2-branch-2.patch,
> HIVE-17472.2.patch, HIVE-17472.3-branch-2.2.patch,
> HIVE-17472.3-branch-2.patch, HIVE-17472.3.patch
>
>
> Raising this on behalf of [~cdrome] and [~selinazh].
> Here's how to reproduce the problem:
> {code:sql}
> CREATE TABLE foobar ( foo STRING, bar STRING ) PARTITIONED BY ( dt STRING,
> region STRING ) STORED AS RCFILE LOCATION '/tmp/foobar';
> ALTER TABLE foobar ADD PARTITION ( dt='1', region='A' ) ;
> dfs -rm -R -skipTrash /tmp/foobar/dt=1;
> ALTER TABLE foobar DROP PARTITION ( dt='1' );
> {code}
> This causes a client-side error as follows:
> {code}
> 15/02/26 23:08:32 ERROR exec.DDLTask:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unknown error. Please check
> logs.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17574) Avoid multiple copies of HDFS-based jars when localizing job-jars
[
https://issues.apache.org/jira/browse/HIVE-17574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17574:
Description:
Raising this on behalf of [~selinazh]. (For my own reference: YHIVE-1035.)
This has to do with the classpaths of Hive actions run from Oozie, and affects
scripts that adds jars/resources from HDFS locations.
As part of Oozie's "sharelib" deploys, foundation jars (such as Hive jars) tend
to be stored in HDFS paths, as are any custom user-libraries used in workflows.
An {{ADD JAR|FILE|ARCHIVE}} statement in a Hive script causes the following
steps to occur:
# Files are downloaded from HDFS to local temp dir.
# UDFs are resolved/validated.
# All jars/files, including those just downloaded from HDFS, are shipped right
back to HDFS-based scratch-directories, for job submission.
For HDFS-based files, this is wasteful and time-consuming. #3 above should skip
shipping HDFS-based resources, and add those directly to the Tez session.
We have a patch that's being used internally at Yahoo.
was:
Raising this on behalf of [~selinazh]. (For my own reference: YHIVE-1035.)
This has to do with the classpaths of Hive actions run from Oozie, and affects
scripts that adds jars/resources from HDFS locations.
As part of Oozie's "sharelib" deploys, foundation jars (such as Hive jars) tend
to be stored in HDFS paths, as are any custom user-libraries used in workflows.
An {{ADD JAR|FILE|ARCHIVE}} statement in a Hive script causes the following
steps to occur:
# Files are downloaded from HDFS to local temp dir.
# UDFs are resolved/validated.
# All jars/files, including those just downloaded from HDFS, are shipped right
back to HDFS-based scratch-directories, for job submission.
This is wasteful and time-consuming. #3 above should skip shipping HDFS-based
resources, and add those directly to the Tez session.
We have a patch that's being used internally at Yahoo.
> Avoid multiple copies of HDFS-based jars when localizing job-jars
> -
>
> Key: HIVE-17574
> URL: https://issues.apache.org/jira/browse/HIVE-17574
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0, 3.0.0, 2.4.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>
> Raising this on behalf of [~selinazh]. (For my own reference: YHIVE-1035.)
> This has to do with the classpaths of Hive actions run from Oozie, and
> affects scripts that adds jars/resources from HDFS locations.
> As part of Oozie's "sharelib" deploys, foundation jars (such as Hive jars)
> tend to be stored in HDFS paths, as are any custom user-libraries used in
> workflows. An {{ADD JAR|FILE|ARCHIVE}} statement in a Hive script causes the
> following steps to occur:
> # Files are downloaded from HDFS to local temp dir.
> # UDFs are resolved/validated.
> # All jars/files, including those just downloaded from HDFS, are shipped
> right back to HDFS-based scratch-directories, for job submission.
> For HDFS-based files, this is wasteful and time-consuming. #3 above should
> skip shipping HDFS-based resources, and add those directly to the Tez session.
> We have a patch that's being used internally at Yahoo.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17570) Fix view deletion related test failures (create_view.q etc)
[
https://issues.apache.org/jira/browse/HIVE-17570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175311#comment-16175311
]
Hive QA commented on HIVE-17570:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888128/HIVE-17570.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11049 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_part_update]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6924/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6924/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6924/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888128 - PreCommit-HIVE-Build
> Fix view deletion related test failures (create_view.q etc)
> ---
>
> Key: HIVE-17570
> URL: https://issues.apache.org/jira/browse/HIVE-17570
> Project: Hive
> Issue Type: Bug
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-17570.1.patch, HIVE-17570.2.patch
>
>
> Fixing the bug introduced by HIVE-17459. Sorry that did not capture that in a
> timely fashion.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17536) StatsUtil::getBasicStatForTable doesn't distinguish b/w absence of statistics or zero stats
[
https://issues.apache.org/jira/browse/HIVE-17536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175312#comment-16175312
]
Hive QA commented on HIVE-17536:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888132/HIVE-17536.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6925/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6925/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6925/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-09-21 19:20:34.754
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-6925/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-09-21 19:20:34.758
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
3e8ae68..afd0d9f master -> origin/master
dec4efd..53dc1ff branch-2 -> origin/branch-2
7b51c96..d95cb29 branch-2.2 -> origin/branch-2.2
+ git reset --hard HEAD
HEAD is now at 3e8ae68 HIVE-17556: The test udf_mask_hash.q is failing (Marta
Kuczora via Zoltan Haindrich)
+ git clean -f -d
Removing standalone-metastore/src/gen/org/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at afd0d9f HIVE-17472: Drop-partition for multi-level partition
fails, if data does not exist. (Chris Drome, reviewed by Mithun Radhakrishnan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-09-21 19:20:38.565
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java: No such
file or directory
error: a/ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out: No
such file or directory
error: a/ql/src/test/results/clientpositive/beeline/smb_mapjoin_1.q.out: No
such file or directory
error: a/ql/src/test/results/clientpositive/beeline/smb_mapjoin_2.q.out: No
such file or directory
error: a/ql/src/test/results/clientpositive/beeline/smb_mapjoin_3.q.out: No
such file or directory
error: a/ql/src/test/results/clientpositive/bucketcontext_5.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/bucketcontext_6.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin5.q.out: No such file
or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out: No
such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out: No
such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin_negative3.q.out: No
such file or directory
error: a/ql/src/test/results/clientpositive/cbo_rp_join1.q.out: No such file or
directory
error:
a/ql/src/test/results/clientpositive/cbo_rp_udaf_percentile_approx_23.q.out: No
such file or directory
error: a/ql/src/test/results/clientpositive/columnstats_tbllvl.q.out: No such
file or directory
error: a/ql/src/test/results/clientpositive/compute_stats_date.q.out: No such
file or directory
error: a/ql/src/test/results/clientpositive/correlationoptimizer5.q.out: No
such file or directory
error: a/ql/src/test/results/clientpositive/decimal_precision.q.out: No such
file or directory
error: a/ql/src/test/results/clientpositive/decimal_udf.q.out: No such file or
directory
error: a/ql/src/test/results/clientpositive/decimal_udf2.q.out: No such file or
directory
error: a/ql/sr
[jira] [Commented] (HIVE-16898) Validation of source file after distcp in repl load
[
https://issues.apache.org/jira/browse/HIVE-16898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175314#comment-16175314
]
Hive QA commented on HIVE-16898:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888328/HIVE-16898.7.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6926/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6926/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6926/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-09-21 19:21:12.083
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-6926/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-09-21 19:21:12.086
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at afd0d9f HIVE-17472: Drop-partition for multi-level partition
fails, if data does not exist. (Chris Drome, reviewed by Mithun Radhakrishnan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at afd0d9f HIVE-17472: Drop-partition for multi-level partition
fails, if data does not exist. (Chris Drome, reviewed by Mithun Radhakrishnan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-09-21 19:21:12.627
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java: No such
file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java: No
such file or directory
error:
a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java:
No such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888328 - PreCommit-HIVE-Build
> Validation of source file after distcp in repl load
>
>
> Key: HIVE-16898
> URL: https://issues.apache.org/jira/browse/HIVE-16898
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: Daniel Dai
> Fix For: 3.0.0
>
> Attachments: HIVE-16898.1.patch, HIVE-16898.2.patch,
> HIVE-16898.3.patch, HIVE-16898.4.patch, HIVE-16898.5.patch,
> HIVE-16898.6.patch, HIVE-16898.7.patch
>
>
> time between deciding the source and destination path for distcp to invoking
> of distcp can have a change of the source file, hence distcp might copy the
> wrong file to destination, hence we should an additional check on the
> checksum of the source file path after distcp finishes to make sure the path
> didnot change during the copy process. if it has take additional steps to
> delete the previous file on destination and copy the new source and repeat
> the same process as above till we copy the correct file.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17402) Provide object location in the HMS notification messages
[
https://issues.apache.org/jira/browse/HIVE-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175315#comment-16175315
]
Hive QA commented on HIVE-17402:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888140/HIVE-17402.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6927/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6927/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6927/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-09-21 19:21:49.483
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-6927/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-09-21 19:21:49.486
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at afd0d9f HIVE-17472: Drop-partition for multi-level partition
fails, if data does not exist. (Chris Drome, reviewed by Mithun Radhakrishnan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at afd0d9f HIVE-17472: Drop-partition for multi-level partition
fails, if data does not exist. (Chris Drome, reviewed by Mithun Radhakrishnan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-09-21 19:21:50.007
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONMessageFactory.java:
No such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888140 - PreCommit-HIVE-Build
> Provide object location in the HMS notification messages
>
>
> Key: HIVE-17402
> URL: https://issues.apache.org/jira/browse/HIVE-17402
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Attachments: HIVE-17402.01.patch
>
>
> While working on the Apache Sentry project that uses HMS notifications we
> noticed that these notifications are using some useful data - e.g. location
> information for the objects. To get around these, ApacheSentry implemented
> its own version of events
> (https://github.com/apache/sentry/tree/master/sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json).
> It seems to be a useful information for Hive as well, so why not add it
> directly into the standard message factory?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17529) Bucket Map Join : Sets incorrect edge type causing execution failure
[ https://issues.apache.org/jira/browse/HIVE-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17529: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master > Bucket Map Join : Sets incorrect edge type causing execution failure > > > Key: HIVE-17529 > URL: https://issues.apache.org/jira/browse/HIVE-17529 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Fix For: 3.0.0 > > Attachments: HIVE-17529.1.patch, HIVE-17529.2.patch, > HIVE-17529.3.patch, HIVE-17529.4.patch, HIVE-17529.5.patch > > > If while traversing the tree to generate tasks, a bucket mapjoin may set its > edge as CUSTOM_SIMPLE_EDGE against CUSTOM_EDGE if the bigtable is already not > traversed causing Tez to assert and fail the vertex. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17110) BucketCodec should enforce value ranges
[
https://issues.apache.org/jira/browse/HIVE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175439#comment-16175439
]
Hive QA commented on HIVE-17110:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888160/HIVE-17110.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11050 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6928/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6928/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6928/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888160 - PreCommit-HIVE-Build
> BucketCodec should enforce value ranges
> ---
>
> Key: HIVE-17110
> URL: https://issues.apache.org/jira/browse/HIVE-17110
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 3.0.0
>
> Attachments: HIVE-17110.01.patch, HIVE-17110.02.patch
>
>
> BucketCodec added in HIVE-16832 can only handle statement id/bucket id values
> in a particular range. Add an assert to check that values are indeed in that
> range.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17536) StatsUtil::getBasicStatForTable doesn't distinguish b/w absence of statistics or zero stats
[ https://issues.apache.org/jira/browse/HIVE-17536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17536: --- Attachment: HIVE-17536.6.patch > StatsUtil::getBasicStatForTable doesn't distinguish b/w absence of statistics > or zero stats > --- > > Key: HIVE-17536 > URL: https://issues.apache.org/jira/browse/HIVE-17536 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17536.1.patch, HIVE-17536.2.patch, > HIVE-17536.3.patch, HIVE-17536.4.patch, HIVE-17536.5.patch, HIVE-17536.6.patch > > > This method returns zero for both of the following cases: > * Statistics are missing in metastore > * Actual stats e.g. number of rows are zero > It'll be good for this method to return e.g. -1 in absence of statistics > instead of assuming it to be zero. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17536) StatsUtil::getBasicStatForTable doesn't distinguish b/w absence of statistics or zero stats
[ https://issues.apache.org/jira/browse/HIVE-17536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17536: --- Status: Open (was: Patch Available) > StatsUtil::getBasicStatForTable doesn't distinguish b/w absence of statistics > or zero stats > --- > > Key: HIVE-17536 > URL: https://issues.apache.org/jira/browse/HIVE-17536 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17536.1.patch, HIVE-17536.2.patch, > HIVE-17536.3.patch, HIVE-17536.4.patch, HIVE-17536.5.patch, HIVE-17536.6.patch > > > This method returns zero for both of the following cases: > * Statistics are missing in metastore > * Actual stats e.g. number of rows are zero > It'll be good for this method to return e.g. -1 in absence of statistics > instead of assuming it to be zero. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17536) StatsUtil::getBasicStatForTable doesn't distinguish b/w absence of statistics or zero stats
[ https://issues.apache.org/jira/browse/HIVE-17536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17536: --- Status: Patch Available (was: Open) > StatsUtil::getBasicStatForTable doesn't distinguish b/w absence of statistics > or zero stats > --- > > Key: HIVE-17536 > URL: https://issues.apache.org/jira/browse/HIVE-17536 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17536.1.patch, HIVE-17536.2.patch, > HIVE-17536.3.patch, HIVE-17536.4.patch, HIVE-17536.5.patch, HIVE-17536.6.patch > > > This method returns zero for both of the following cases: > * Statistics are missing in metastore > * Actual stats e.g. number of rows are zero > It'll be good for this method to return e.g. -1 in absence of statistics > instead of assuming it to be zero. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17575) main ERROR Unable to locate appender "FA" for logger config "root"
[
https://issues.apache.org/jira/browse/HIVE-17575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175490#comment-16175490
]
mohammed morshed commented on HIVE-17575:
-
Below is the Workaround to resolve the issue:
===
In your Hadoop cluster nodes, edit the file
“/etc/hive/conf.dist/hive-exec-log4j2.properties”
property.hive.root.logger = FA
appender.FA.type = RandomAccessFile
appender.FA.name = FA
appender.FA.fileName = ${sys:hive.log.dir}/${sys:hive.log.file1}
and
Also edit in your cluster nodes “/etc/hadoop/conf.empty/log4j.properties”
hadoop.log.file1=hadoop1.log
hadoop.log.file2=hadoop2.log
hadoop.log.file3=hadoop3.log
log4j.appender.HADOOP.File=${hadoop.log.dir}/${hadoop.log.file1}
log4j.appender.MAPRED.File=${hadoop.log.dir}/${hadoop.log.file2}
log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file3}
+++
> main ERROR Unable to locate appender "FA" for logger config "root"
> --
>
> Key: HIVE-17575
> URL: https://issues.apache.org/jira/browse/HIVE-17575
> Project: Hive
> Issue Type: Bug
> Components: Beeline, Hive
>Reporter: mohammed morshed
>Priority: Critical
> Labels: usability
>
> Problem: In Hive version 2.3, when running 'INSERT' statements from beeline,
> observed following error as below:
> 0: jdbc:hive2://localhost:1>INSERT INTO TABLE
> w2867998436858995169_write_orders_bkt_tgt_tmp_m_orders_updtx_50percent SELECT
> orders_bkt.o_orderkey as a0, orders_bkt.o_custkey as a1,
> orders_bkt.o_totalprice as a2, orders_bkt.o_orderdate as a3,
> orders_bkt.o_orderpriority as a4, orders_bkt.o_clerk as a5,
> orders_bkt.o_shippriority as a6, orders_bkt.o_comment as a7,
> orders_bkt.o_orderstatus as a8 FROM
> w2867998436858995169_write_orders_bkt_src_tmp_m_orders_updtx_50percent JOIN
> TPCH_TEXT_S3_SINGLE_100.orders_bkt ON
> (w2867998436858995169_write_orders_bkt_src_tmp_m_orders_updtx_50percent.a0 =
> orders_bkt.o_orderkey);
> WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. spark,
> tez) or using Hive 1.X releases.
> INFO : WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available
> in the future versions. Consider using a different execution engine (i.e.
> spark, tez) or using Hive 1.X releases.
> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. spark,
> tez) or using Hive 1.X releases.
> INFO : Query ID = hive_20170921204456_1d837547-fa86-43c5-b57f-e16085abc5d8
> INFO : Total jobs = 3
> INFO : Starting task [Stage-6:CONDITIONAL] in serial mode
> INFO : Stage-7 is selected by condition resolver.
> INFO : Stage-1 is filtered out by condition resolver.
> INFO : Starting task [Stage-7:MAPREDLOCAL] in serial mode
> 2017-09-21 20:46:54,822 main ERROR Unable to invoke factory method in class
> class org.apache.logging.log4j.core.appender.RandomAccessFileAppender for
> element RandomAccessFile. java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:132)
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:918)
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:858)
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:850)
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:479)
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:219)
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:231)
> at
> org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:496)
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:566)
> at
> org.apache.logging.log4j.core.LoggerContext.setConfigLocation(LoggerContext.java:555)
> at
> org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.locateContext(ClassLoaderContextSelector.java:157)
> at
> org.apache.log
[jira] [Commented] (HIVE-14813) Make TransactionBatchImpl.toString() include state of each txn: commit/abort
[
https://issues.apache.org/jira/browse/HIVE-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175540#comment-16175540
]
Hive QA commented on HIVE-14813:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888159/HIVE-14813.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11050 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_part_update]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6929/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6929/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6929/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888159 - PreCommit-HIVE-Build
> Make TransactionBatchImpl.toString() include state of each txn: commit/abort
>
>
> Key: HIVE-14813
> URL: https://issues.apache.org/jira/browse/HIVE-14813
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, Transactions
>Affects Versions: 1.2.1
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14813.01.patch
>
>
> perhaps include user abort vs system abort on batch close
> also include if close() was called and if there was an exception
> cc [~deepesh]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17529) Bucket Map Join : Sets incorrect edge type causing execution failure
[ https://issues.apache.org/jira/browse/HIVE-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175569#comment-16175569 ] Deepak Jaiswal commented on HIVE-17529: --- Thanks for the review and commit [~jdere] > Bucket Map Join : Sets incorrect edge type causing execution failure > > > Key: HIVE-17529 > URL: https://issues.apache.org/jira/browse/HIVE-17529 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Fix For: 3.0.0 > > Attachments: HIVE-17529.1.patch, HIVE-17529.2.patch, > HIVE-17529.3.patch, HIVE-17529.4.patch, HIVE-17529.5.patch > > > If while traversing the tree to generate tasks, a bucket mapjoin may set its > edge as CUSTOM_SIMPLE_EDGE against CUSTOM_EDGE if the bigtable is already not > traversed causing Tez to assert and fail the vertex. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17489) Separate client-facing and server-side Kerberos principals, to support HA
[
https://issues.apache.org/jira/browse/HIVE-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175615#comment-16175615
]
Hive QA commented on HIVE-17489:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888169/HIVE-17489.3-branch-2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10591 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=153)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[merge_negative_5]
(batchId=88)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=99)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf]
(batchId=125)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=176)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData[5]
(batchId=179)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
(batchId=237)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6930/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6930/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6930/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888169 - PreCommit-HIVE-Build
> Separate client-facing and server-side Kerberos principals, to support HA
> -
>
> Key: HIVE-17489
> URL: https://issues.apache.org/jira/browse/HIVE-17489
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Mithun Radhakrishnan
>Assignee: Thiruvel Thirumoolan
> Attachments: HIVE-17489.2-branch-2.patch, HIVE-17489.2.patch,
> HIVE-17489.2.patch, HIVE-17489.3-branch-2.patch, HIVE-17489.3.patch
>
>
> On deployments of the Hive metastore where a farm of servers is fronted by a
> VIP, the hostname of the VIP (e.g. {{mycluster-hcat.blue.myth.net}}) will
> differ from the actual boxen in the farm (.e.g
> {{mycluster-hcat-\[0..3\].blue.myth.net}}).
> Such a deployment messes up Kerberos auth, with principals like
> {{hcat/mycluster-hcat.blue.myth@grid.myth.net}}. Host-based checks will
> disallow servers behind the VIP from using the VIP's hostname in its
> principal when accessing, say, HDFS.
> The solution would be to decouple the server-side principal (used to access
> other services like HDFS as a client) from the client-facing principal (used
> from Hive-client, BeeLine, etc.).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17574) Avoid multiple copies of HDFS-based jars when localizing job-jars
[
https://issues.apache.org/jira/browse/HIVE-17574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17574:
Attachment: HIVE-17574.1-branch-2.2.patch
HIVE-17574.1-branch-2.patch
HIVE-17574.1.patch
The proposed fix, for 3 branches.
> Avoid multiple copies of HDFS-based jars when localizing job-jars
> -
>
> Key: HIVE-17574
> URL: https://issues.apache.org/jira/browse/HIVE-17574
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0, 3.0.0, 2.4.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-17574.1-branch-2.2.patch,
> HIVE-17574.1-branch-2.patch, HIVE-17574.1.patch
>
>
> Raising this on behalf of [~selinazh]. (For my own reference: YHIVE-1035.)
> This has to do with the classpaths of Hive actions run from Oozie, and
> affects scripts that adds jars/resources from HDFS locations.
> As part of Oozie's "sharelib" deploys, foundation jars (such as Hive jars)
> tend to be stored in HDFS paths, as are any custom user-libraries used in
> workflows. An {{ADD JAR|FILE|ARCHIVE}} statement in a Hive script causes the
> following steps to occur:
> # Files are downloaded from HDFS to local temp dir.
> # UDFs are resolved/validated.
> # All jars/files, including those just downloaded from HDFS, are shipped
> right back to HDFS-based scratch-directories, for job submission.
> For HDFS-based files, this is wasteful and time-consuming. #3 above should
> skip shipping HDFS-based resources, and add those directly to the Tez session.
> We have a patch that's being used internally at Yahoo.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17574) Avoid multiple copies of HDFS-based jars when localizing job-jars
[
https://issues.apache.org/jira/browse/HIVE-17574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan reassigned HIVE-17574:
---
Assignee: Chris Drome (was: Mithun Radhakrishnan)
> Avoid multiple copies of HDFS-based jars when localizing job-jars
> -
>
> Key: HIVE-17574
> URL: https://issues.apache.org/jira/browse/HIVE-17574
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0, 3.0.0, 2.4.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17574.1-branch-2.2.patch,
> HIVE-17574.1-branch-2.patch, HIVE-17574.1.patch
>
>
> Raising this on behalf of [~selinazh]. (For my own reference: YHIVE-1035.)
> This has to do with the classpaths of Hive actions run from Oozie, and
> affects scripts that adds jars/resources from HDFS locations.
> As part of Oozie's "sharelib" deploys, foundation jars (such as Hive jars)
> tend to be stored in HDFS paths, as are any custom user-libraries used in
> workflows. An {{ADD JAR|FILE|ARCHIVE}} statement in a Hive script causes the
> following steps to occur:
> # Files are downloaded from HDFS to local temp dir.
> # UDFs are resolved/validated.
> # All jars/files, including those just downloaded from HDFS, are shipped
> right back to HDFS-based scratch-directories, for job submission.
> For HDFS-based files, this is wasteful and time-consuming. #3 above should
> skip shipping HDFS-based resources, and add those directly to the Tez session.
> We have a patch that's being used internally at Yahoo.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17574) Avoid multiple copies of HDFS-based jars when localizing job-jars
[
https://issues.apache.org/jira/browse/HIVE-17574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17574:
Status: Patch Available (was: Open)
> Avoid multiple copies of HDFS-based jars when localizing job-jars
> -
>
> Key: HIVE-17574
> URL: https://issues.apache.org/jira/browse/HIVE-17574
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0, 3.0.0, 2.4.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17574.1-branch-2.2.patch,
> HIVE-17574.1-branch-2.patch, HIVE-17574.1.patch
>
>
> Raising this on behalf of [~selinazh]. (For my own reference: YHIVE-1035.)
> This has to do with the classpaths of Hive actions run from Oozie, and
> affects scripts that adds jars/resources from HDFS locations.
> As part of Oozie's "sharelib" deploys, foundation jars (such as Hive jars)
> tend to be stored in HDFS paths, as are any custom user-libraries used in
> workflows. An {{ADD JAR|FILE|ARCHIVE}} statement in a Hive script causes the
> following steps to occur:
> # Files are downloaded from HDFS to local temp dir.
> # UDFs are resolved/validated.
> # All jars/files, including those just downloaded from HDFS, are shipped
> right back to HDFS-based scratch-directories, for job submission.
> For HDFS-based files, this is wasteful and time-consuming. #3 above should
> skip shipping HDFS-based resources, and add those directly to the Tez session.
> We have a patch that's being used internally at Yahoo.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17563) CodahaleMetrics.JsonFileReporter is not updating hive.service.metrics.file.location
[
https://issues.apache.org/jira/browse/HIVE-17563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-17563:
--
Attachment: HIVE-17563.01.patch
> CodahaleMetrics.JsonFileReporter is not updating
> hive.service.metrics.file.location
> ---
>
> Key: HIVE-17563
> URL: https://issues.apache.org/jira/browse/HIVE-17563
> Project: Hive
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Alexander Kolbasov
> Attachments: HIVE-17563.01.patch
>
>
> The {{JsonFileReporter}} has a bug where it never updates the content of
> {{hive.service.metrics.file.location}}. The original implementation relied on
> {{fs.rename}} to overwrite the file if it already existed, but after
> HIVE-13705 {{fs.rename}} now returns {{false}} if the destination file exists.
> Also, there is no reason for the {{JsonFileReporter}} to write to any fs
> besides the local filesystem, so we should make this local fs specific.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17563) CodahaleMetrics.JsonFileReporter is not updating hive.service.metrics.file.location
[
https://issues.apache.org/jira/browse/HIVE-17563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-17563:
--
Status: Patch Available (was: Open)
> CodahaleMetrics.JsonFileReporter is not updating
> hive.service.metrics.file.location
> ---
>
> Key: HIVE-17563
> URL: https://issues.apache.org/jira/browse/HIVE-17563
> Project: Hive
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Alexander Kolbasov
> Attachments: HIVE-17563.01.patch
>
>
> The {{JsonFileReporter}} has a bug where it never updates the content of
> {{hive.service.metrics.file.location}}. The original implementation relied on
> {{fs.rename}} to overwrite the file if it already existed, but after
> HIVE-13705 {{fs.rename}} now returns {{false}} if the destination file exists.
> Also, there is no reason for the {{JsonFileReporter}} to write to any fs
> besides the local filesystem, so we should make this local fs specific.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17543) Enable PerfCliDriver for HoS
[ https://issues.apache.org/jira/browse/HIVE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-17543: --- Assignee: Sahil Takiar > Enable PerfCliDriver for HoS > > > Key: HIVE-17543 > URL: https://issues.apache.org/jira/browse/HIVE-17543 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar > > The PerfCliDriver contains .q files for TPC-DS queries. It doesn't actually > run them, but it does generate explains for them. It also tricks HMS into > thinking its a 30 TB TPC-DS dataset so that the explain plan triggers certain > optimizations. > Right now this only runs of Hive-on-Tez, we should enable it for HoS too. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17563) CodahaleMetrics.JsonFileReporter is not updating hive.service.metrics.file.location
[
https://issues.apache.org/jira/browse/HIVE-17563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175678#comment-16175678
]
Alexander Kolbasov commented on HIVE-17563:
---
As part of the changes I am also adding a few things:
- The reporter now has a useful thread name
- Some cleanup for TestCodahaleMetrics
> CodahaleMetrics.JsonFileReporter is not updating
> hive.service.metrics.file.location
> ---
>
> Key: HIVE-17563
> URL: https://issues.apache.org/jira/browse/HIVE-17563
> Project: Hive
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Alexander Kolbasov
> Attachments: HIVE-17563.01.patch
>
>
> The {{JsonFileReporter}} has a bug where it never updates the content of
> {{hive.service.metrics.file.location}}. The original implementation relied on
> {{fs.rename}} to overwrite the file if it already existed, but after
> HIVE-13705 {{fs.rename}} now returns {{false}} if the destination file exists.
> Also, there is no reason for the {{JsonFileReporter}} to write to any fs
> besides the local filesystem, so we should make this local fs specific.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17563) CodahaleMetrics.JsonFileReporter is not updating hive.service.metrics.file.location
[
https://issues.apache.org/jira/browse/HIVE-17563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175681#comment-16175681
]
Alexander Kolbasov commented on HIVE-17563:
---
There is another reporter that is potentially affected:
{{org.apache.hadoop.hive.metastore.metrics.JsonReporter}} - it should be
updated as well.
> CodahaleMetrics.JsonFileReporter is not updating
> hive.service.metrics.file.location
> ---
>
> Key: HIVE-17563
> URL: https://issues.apache.org/jira/browse/HIVE-17563
> Project: Hive
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Alexander Kolbasov
> Attachments: HIVE-17563.01.patch
>
>
> The {{JsonFileReporter}} has a bug where it never updates the content of
> {{hive.service.metrics.file.location}}. The original implementation relied on
> {{fs.rename}} to overwrite the file if it already existed, but after
> HIVE-13705 {{fs.rename}} now returns {{false}} if the destination file exists.
> Also, there is no reason for the {{JsonFileReporter}} to write to any fs
> besides the local filesystem, so we should make this local fs specific.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17576) Improve progress-reporting in TezProcessor
[
https://issues.apache.org/jira/browse/HIVE-17576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan reassigned HIVE-17576:
---
> Improve progress-reporting in TezProcessor
> --
>
> Key: HIVE-17576
> URL: https://issues.apache.org/jira/browse/HIVE-17576
> Project: Hive
> Issue Type: Bug
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
>
> Another one on behalf of [~selinazh] and [~cdrome]. Following the example in
> [Apache Tez's
> {{MapProcessor}}|https://github.com/apache/tez/blob/247719d7314232f680f028f4e1a19370ffb7b1bb/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/map/MapProcessor.java#L88],
> {{TezProcessor}} ought to use {{ProgressHelper}} to report progress for a
> Tez task. As per [~kshukla]'s advice,
> {quote}
> Tez... provides {{getProgress()}} API for {{AbstractLogicalInput(s)}} which
> will give the correct progress value for a given Input. The TezProcessor(s)
> in Hive should use this to do something similar to what MapProcessor in Tez
> does today, which is use/override ProgressHelper to get the input progress
> and then set the progress on the processorContext.
> ...
> The default behavior of the ProgressHelper class sets the processor progress
> to be the average of progress values from all inputs.
> {quote}
> This code is -whacked from- *inspired by* {{MapProcessor}}'s use of
> {{ProgressHelper}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17576) Improve progress-reporting in TezProcessor
[
https://issues.apache.org/jira/browse/HIVE-17576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17576:
Description:
Another one on behalf of [~selinazh] and [~cdrome]. Following the example in
[Apache Tez's
{{MapProcessor}}|https://github.com/apache/tez/blob/247719d7314232f680f028f4e1a19370ffb7b1bb/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/map/MapProcessor.java#L88],
{{TezProcessor}} ought to use {{ProgressHelper}} to report progress for a Tez
task. As per [~kshukla]'s advice,
{quote}
Tez... provides {{getProgress()}} API for {{AbstractLogicalInput(s)}} which
will give the correct progress value for a given Input. The TezProcessor(s) in
Hive should use this to do something similar to what MapProcessor in Tez does
today, which is use/override ProgressHelper to get the input progress and then
set the progress on the processorContext.
...
The default behavior of the ProgressHelper class sets the processor progress to
be the average of progress values from all inputs.
{quote}
This code is -whacked from- *inspired by* {{MapProcessor}}'s use of
{{ProgressHelper}}.
(For my reference, YHIVE-978.)
was:
Another one on behalf of [~selinazh] and [~cdrome]. Following the example in
[Apache Tez's
{{MapProcessor}}|https://github.com/apache/tez/blob/247719d7314232f680f028f4e1a19370ffb7b1bb/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/map/MapProcessor.java#L88],
{{TezProcessor}} ought to use {{ProgressHelper}} to report progress for a Tez
task. As per [~kshukla]'s advice,
{quote}
Tez... provides {{getProgress()}} API for {{AbstractLogicalInput(s)}} which
will give the correct progress value for a given Input. The TezProcessor(s) in
Hive should use this to do something similar to what MapProcessor in Tez does
today, which is use/override ProgressHelper to get the input progress and then
set the progress on the processorContext.
...
The default behavior of the ProgressHelper class sets the processor progress to
be the average of progress values from all inputs.
{quote}
This code is -whacked from- *inspired by* {{MapProcessor}}'s use of
{{ProgressHelper}}.
> Improve progress-reporting in TezProcessor
> --
>
> Key: HIVE-17576
> URL: https://issues.apache.org/jira/browse/HIVE-17576
> Project: Hive
> Issue Type: Bug
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
>
> Another one on behalf of [~selinazh] and [~cdrome]. Following the example in
> [Apache Tez's
> {{MapProcessor}}|https://github.com/apache/tez/blob/247719d7314232f680f028f4e1a19370ffb7b1bb/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/map/MapProcessor.java#L88],
> {{TezProcessor}} ought to use {{ProgressHelper}} to report progress for a
> Tez task. As per [~kshukla]'s advice,
> {quote}
> Tez... provides {{getProgress()}} API for {{AbstractLogicalInput(s)}} which
> will give the correct progress value for a given Input. The TezProcessor(s)
> in Hive should use this to do something similar to what MapProcessor in Tez
> does today, which is use/override ProgressHelper to get the input progress
> and then set the progress on the processorContext.
> ...
> The default behavior of the ProgressHelper class sets the processor progress
> to be the average of progress values from all inputs.
> {quote}
> This code is -whacked from- *inspired by* {{MapProcessor}}'s use of
> {{ProgressHelper}}.
> (For my reference, YHIVE-978.)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17576) Improve progress-reporting in TezProcessor
[
https://issues.apache.org/jira/browse/HIVE-17576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175702#comment-16175702
]
Mithun Radhakrishnan commented on HIVE-17576:
-
Hmm... Looks like the {{ProgressHelper}} is only available in Tez 0.9. This
might need to go only to {{master}}. :/
> Improve progress-reporting in TezProcessor
> --
>
> Key: HIVE-17576
> URL: https://issues.apache.org/jira/browse/HIVE-17576
> Project: Hive
> Issue Type: Bug
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
>
> Another one on behalf of [~selinazh] and [~cdrome]. Following the example in
> [Apache Tez's
> {{MapProcessor}}|https://github.com/apache/tez/blob/247719d7314232f680f028f4e1a19370ffb7b1bb/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/map/MapProcessor.java#L88],
> {{TezProcessor}} ought to use {{ProgressHelper}} to report progress for a
> Tez task. As per [~kshukla]'s advice,
> {quote}
> Tez... provides {{getProgress()}} API for {{AbstractLogicalInput(s)}} which
> will give the correct progress value for a given Input. The TezProcessor(s)
> in Hive should use this to do something similar to what MapProcessor in Tez
> does today, which is use/override ProgressHelper to get the input progress
> and then set the progress on the processorContext.
> ...
> The default behavior of the ProgressHelper class sets the processor progress
> to be the average of progress values from all inputs.
> {quote}
> This code is -whacked from- *inspired by* {{MapProcessor}}'s use of
> {{ProgressHelper}}.
> (For my reference, YHIVE-978.)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17576) Improve progress-reporting in TezProcessor
[
https://issues.apache.org/jira/browse/HIVE-17576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17576:
Attachment: HIVE-17576.1.patch
Patch for {{master}}.
> Improve progress-reporting in TezProcessor
> --
>
> Key: HIVE-17576
> URL: https://issues.apache.org/jira/browse/HIVE-17576
> Project: Hive
> Issue Type: Bug
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17576.1.patch
>
>
> Another one on behalf of [~selinazh] and [~cdrome]. Following the example in
> [Apache Tez's
> {{MapProcessor}}|https://github.com/apache/tez/blob/247719d7314232f680f028f4e1a19370ffb7b1bb/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/map/MapProcessor.java#L88],
> {{TezProcessor}} ought to use {{ProgressHelper}} to report progress for a
> Tez task. As per [~kshukla]'s advice,
> {quote}
> Tez... provides {{getProgress()}} API for {{AbstractLogicalInput(s)}} which
> will give the correct progress value for a given Input. The TezProcessor(s)
> in Hive should use this to do something similar to what MapProcessor in Tez
> does today, which is use/override ProgressHelper to get the input progress
> and then set the progress on the processorContext.
> ...
> The default behavior of the ProgressHelper class sets the processor progress
> to be the average of progress values from all inputs.
> {quote}
> This code is -whacked from- *inspired by* {{MapProcessor}}'s use of
> {{ProgressHelper}}.
> (For my reference, YHIVE-978.)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17576) Improve progress-reporting in TezProcessor
[
https://issues.apache.org/jira/browse/HIVE-17576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17576:
Status: Patch Available (was: Open)
Submitting for tests.
> Improve progress-reporting in TezProcessor
> --
>
> Key: HIVE-17576
> URL: https://issues.apache.org/jira/browse/HIVE-17576
> Project: Hive
> Issue Type: Bug
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17576.1.patch
>
>
> Another one on behalf of [~selinazh] and [~cdrome]. Following the example in
> [Apache Tez's
> {{MapProcessor}}|https://github.com/apache/tez/blob/247719d7314232f680f028f4e1a19370ffb7b1bb/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/map/MapProcessor.java#L88],
> {{TezProcessor}} ought to use {{ProgressHelper}} to report progress for a
> Tez task. As per [~kshukla]'s advice,
> {quote}
> Tez... provides {{getProgress()}} API for {{AbstractLogicalInput(s)}} which
> will give the correct progress value for a given Input. The TezProcessor(s)
> in Hive should use this to do something similar to what MapProcessor in Tez
> does today, which is use/override ProgressHelper to get the input progress
> and then set the progress on the processorContext.
> ...
> The default behavior of the ProgressHelper class sets the processor progress
> to be the average of progress values from all inputs.
> {quote}
> This code is -whacked from- *inspired by* {{MapProcessor}}'s use of
> {{ProgressHelper}}.
> (For my reference, YHIVE-978.)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17576) Improve progress-reporting in TezProcessor
[
https://issues.apache.org/jira/browse/HIVE-17576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17576:
Affects Version/s: 2.4.0
3.0.0
2.2.0
> Improve progress-reporting in TezProcessor
> --
>
> Key: HIVE-17576
> URL: https://issues.apache.org/jira/browse/HIVE-17576
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0, 3.0.0, 2.4.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17576.1.patch
>
>
> Another one on behalf of [~selinazh] and [~cdrome]. Following the example in
> [Apache Tez's
> {{MapProcessor}}|https://github.com/apache/tez/blob/247719d7314232f680f028f4e1a19370ffb7b1bb/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/map/MapProcessor.java#L88],
> {{TezProcessor}} ought to use {{ProgressHelper}} to report progress for a
> Tez task. As per [~kshukla]'s advice,
> {quote}
> Tez... provides {{getProgress()}} API for {{AbstractLogicalInput(s)}} which
> will give the correct progress value for a given Input. The TezProcessor(s)
> in Hive should use this to do something similar to what MapProcessor in Tez
> does today, which is use/override ProgressHelper to get the input progress
> and then set the progress on the processorContext.
> ...
> The default behavior of the ProgressHelper class sets the processor progress
> to be the average of progress values from all inputs.
> {quote}
> This code is -whacked from- *inspired by* {{MapProcessor}}'s use of
> {{ProgressHelper}}.
> (For my reference, YHIVE-978.)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17565) NullPointerException occurs when hive.optimize.skewjoin and hive.auto.convert.join are switched on at the same time
[ https://issues.apache.org/jira/browse/HIVE-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175715#comment-16175715 ] Xin Hao commented on HIVE-17565: Hive on MR. Thanks. > NullPointerException occurs when hive.optimize.skewjoin and > hive.auto.convert.join are switched on at the same time > --- > > Key: HIVE-17565 > URL: https://issues.apache.org/jira/browse/HIVE-17565 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Xin Hao >Assignee: liyunzhang_intel > > (A)NullPointerException occurs when hive.optimize.skewjoin and > hive.auto.convert.join are switched on at the same time. > Could pass when hive.optimize.skewjoin=true and hive.auto.convert.join=false. > (B)Hive Version: > Found on Apache Hive 1.2.1 > (C)Workload: > (1)TPCx-BB Q19 > (2) A small case as below，which was actually simplified from Q19： > SELECT * > FROM store_returns sr, > ( > SELECT d1.d_date_sk > FROM date_dim d1, date_dim d2 > WHERE d1.d_week_seq = d2.d_week_seq > ) sr_dateFilter > WHERE sr.sr_returned_date_sk = d_date_sk; > (D)Exception Error Message: > Error: java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:194) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:223) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1051) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055) > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:490) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170) > ... 8 more -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-17558: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Thanks for the patch Tao, and for the review Daniel! > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Fix For: 3.0.0 > > Attachments: HIVE-17558.1.patch, HIVE-17558.2.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17543) Enable PerfCliDriver for HoS
[ https://issues.apache.org/jira/browse/HIVE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-17543: Attachment: HIVE-17543.1.patch > Enable PerfCliDriver for HoS > > > Key: HIVE-17543 > URL: https://issues.apache.org/jira/browse/HIVE-17543 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-17543.1.patch > > > The PerfCliDriver contains .q files for TPC-DS queries. It doesn't actually > run them, but it does generate explains for them. It also tricks HMS into > thinking its a 30 TB TPC-DS dataset so that the explain plan triggers certain > optimizations. > Right now this only runs of Hive-on-Tez, we should enable it for HoS too. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17543) Enable PerfCliDriver for HoS
[ https://issues.apache.org/jira/browse/HIVE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-17543: Status: Patch Available (was: Open) > Enable PerfCliDriver for HoS > > > Key: HIVE-17543 > URL: https://issues.apache.org/jira/browse/HIVE-17543 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-17543.1.patch > > > The PerfCliDriver contains .q files for TPC-DS queries. It doesn't actually > run them, but it does generate explains for them. It also tricks HMS into > thinking its a 30 TB TPC-DS dataset so that the explain plan triggers certain > optimizations. > Right now this only runs of Hive-on-Tez, we should enable it for HoS too. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17568) HiveJoinPushTransitivePredicatesRule may exchange predicates which are not valid on the other branch
[
https://issues.apache.org/jira/browse/HIVE-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175736#comment-16175736
]
Hive QA commented on HIVE-17568:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888226/HIVE-17568.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11051 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_semijoin]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_semijoin]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6931/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6931/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6931/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888226 - PreCommit-HIVE-Build
> HiveJoinPushTransitivePredicatesRule may exchange predicates which are not
> valid on the other branch
>
>
> Key: HIVE-17568
> URL: https://issues.apache.org/jira/browse/HIVE-17568
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-17568.01.patch, HIVE-17568.02.patch
>
>
> Joining 2 tables on at least 1 column which is not the same type ;
> (integer/double for example).
> The calcite expressions require double/integer inputs which will became
> invalid if {{HiveJoinPushTransitivePredicatesRule}} pushes them to the other
> branch.
> query:
> {code}
> create table t1 (v string, k int);
> insert into t1 values ('people', 10), ('strangers', 20), ('parents', 30);
> create table t2 (v string, k double);
> insert into t2 values ('people', 10), ('strangers', 20), ('parents', 30);
> select * from t1 where t1.k in (select t2.k from t2 where t2.v='people') and
> t1.k<15;
> {code}
> results in:
> {code}
> java.lang.AssertionError: type mismatch:
> type1:
> DOUBLE
> type2:
> INTEGER
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:1841)
> at
> org.apache.calcite.rex.RexProgramBuilder$RegisterInputShuttle.visitInputRef(RexProgramBuilder.java:941)
> at
> org.apache.calcite.rex.RexProgramBuilder$RegisterInputShuttle.visitInputRef(RexProgramBuilder.java:919)
> at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
> at org.apache.calcite.rex.RexShuttle.visitList(RexShuttle.java:153)
> at org.apache.calcite.rex.RexShuttle.visitCall(RexShuttle.java:102)
> at
> org.apache.calcite.rex.RexProgramBuilder$RegisterShuttle.visitCall(RexProgramBuilder.java:884)
> at
> org.apache.calcite.rex.RexProgramBuilder$RegisterShuttle.visitCall(RexProgramBuilder.java:882)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:104)
> at
> org.apache.calcite.rex.RexProgramBuilder.registerInput(RexProgramBuilder.java:296)
> at
> org.apache.calcite.rex.RexProgramBuilder.addCondition(RexProgramBuilder.java:271)
> at
> org.apache.calcite.rel.rules.FilterMergeRule.createProgram(FilterMergeRule.java:98)
> at
> org.apache.calcite.rel.rules.FilterMergeRule.onMatch(FilterMergeRule.java:67)
> [...]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17402) Provide object location in the HMS notification messages
[ https://issues.apache.org/jira/browse/HIVE-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17402: -- Attachment: HIVE-17402.01.patch > Provide object location in the HMS notification messages > > > Key: HIVE-17402 > URL: https://issues.apache.org/jira/browse/HIVE-17402 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > Attachments: HIVE-17402.01.patch > > > While working on the Apache Sentry project that uses HMS notifications we > noticed that these notifications are using some useful data - e.g. location > information for the objects. To get around these, ApacheSentry implemented > its own version of events > (https://github.com/apache/sentry/tree/master/sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json). > It seems to be a useful information for Hive as well, so why not add it > directly into the standard message factory? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17402) Provide object location in the HMS notification messages
[ https://issues.apache.org/jira/browse/HIVE-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17402: -- Status: Open (was: Patch Available) > Provide object location in the HMS notification messages > > > Key: HIVE-17402 > URL: https://issues.apache.org/jira/browse/HIVE-17402 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > Attachments: HIVE-17402.01.patch > > > While working on the Apache Sentry project that uses HMS notifications we > noticed that these notifications are using some useful data - e.g. location > information for the objects. To get around these, ApacheSentry implemented > its own version of events > (https://github.com/apache/sentry/tree/master/sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json). > It seems to be a useful information for Hive as well, so why not add it > directly into the standard message factory? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17402) Provide object location in the HMS notification messages
[ https://issues.apache.org/jira/browse/HIVE-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17402: -- Status: Patch Available (was: Open) > Provide object location in the HMS notification messages > > > Key: HIVE-17402 > URL: https://issues.apache.org/jira/browse/HIVE-17402 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > Attachments: HIVE-17402.01.patch > > > While working on the Apache Sentry project that uses HMS notifications we > noticed that these notifications are using some useful data - e.g. location > information for the objects. To get around these, ApacheSentry implemented > its own version of events > (https://github.com/apache/sentry/tree/master/sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json). > It seems to be a useful information for Hive as well, so why not add it > directly into the standard message factory? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17402) Provide object location in the HMS notification messages
[ https://issues.apache.org/jira/browse/HIVE-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17402: -- Attachment: (was: HIVE-17402.01.patch) > Provide object location in the HMS notification messages > > > Key: HIVE-17402 > URL: https://issues.apache.org/jira/browse/HIVE-17402 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > Attachments: HIVE-17402.01.patch > > > While working on the Apache Sentry project that uses HMS notifications we > noticed that these notifications are using some useful data - e.g. location > information for the objects. To get around these, ApacheSentry implemented > its own version of events > (https://github.com/apache/sentry/tree/master/sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json). > It seems to be a useful information for Hive as well, so why not add it > directly into the standard message factory? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17563) CodahaleMetrics.JsonFileReporter is not updating hive.service.metrics.file.location
[
https://issues.apache.org/jira/browse/HIVE-17563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175790#comment-16175790
]
Alexander Kolbasov commented on HIVE-17563:
---
There is an interesting side effect of using the current code - it also creates
.crc files for metric which are completely useless, because by default it uses
the LocalFileSystem which adds crc.
> CodahaleMetrics.JsonFileReporter is not updating
> hive.service.metrics.file.location
> ---
>
> Key: HIVE-17563
> URL: https://issues.apache.org/jira/browse/HIVE-17563
> Project: Hive
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Alexander Kolbasov
> Attachments: HIVE-17563.01.patch
>
>
> The {{JsonFileReporter}} has a bug where it never updates the content of
> {{hive.service.metrics.file.location}}. The original implementation relied on
> {{fs.rename}} to overwrite the file if it already existed, but after
> HIVE-13705 {{fs.rename}} now returns {{false}} if the destination file exists.
> Also, there is no reason for the {{JsonFileReporter}} to write to any fs
> besides the local filesystem, so we should make this local fs specific.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[
https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175798#comment-16175798
]
Hive QA commented on HIVE-17139:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888297/HIVE-17139.12.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 11051 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_when_case]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
(batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6932/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6932/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6932/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888297 - PreCommit-HIVE-Build
> Conditional expressions optimization: skip the expression evaluation if the
> condition is not satisfied for vectorization engine.
>
>
> Key: HIVE-17139
> URL: https://issues.apache.org/jira/browse/HIVE-17139
> Project: Hive
> Issue Type: Improvement
>Reporter: Ke Jia
>Assignee: Ke Jia
> Attachments: HIVE-17139.10.patch, HIVE-17139.11.patch,
> HIVE-17139.12.patch, HIVE-17139.1.patch, HIVE-17139.2.patch,
> HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch,
> HIVE-17139.6.patch, HIVE-17139.7.patch, HIVE-17139.8.patch, HIVE-17139.9.patch
>
>
> The case when and if statement execution for Hive vectorization is not
> optimal, which all the conditional and else expressions are evaluated for
> current implementation. The optimized approach is to update the selected
> array of batch parameter after the conditional expression is executed. Then
> the else expression will only do the selected rows instead of all.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16898) Validation of source file after distcp in repl load
[
https://issues.apache.org/jira/browse/HIVE-16898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175835#comment-16175835
]
Hive QA commented on HIVE-16898:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888328/HIVE-16898.7.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6934/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6934/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6934/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-09-22 03:19:44.041
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-6934/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-09-22 03:19:44.044
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 266c505 HIVE-17570 : Fix view deletion related test failures
(create_view.q etc) (Tao Li via Thejas Nair)
+ git clean -f -d
Removing
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/security/
Removing
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java.orig
Removing
service/src/java/org/apache/hive/service/auth/HiveDelegationTokenManager.java
Removing
service/src/java/org/apache/hive/service/auth/HiveTokenStoreDelegationTokenSecretManager.java
Removing standalone-metastore/src/gen/org/
Removing
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/DBTokenStore.java
Removing
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/ZooKeeperTokenStore.java
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 266c505 HIVE-17570 : Fix view deletion related test failures
(create_view.q etc) (Tao Li via Thejas Nair)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-09-22 03:19:44.955
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java: No such
file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java: No
such file or directory
error:
a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java:
No such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888328 - PreCommit-HIVE-Build
> Validation of source file after distcp in repl load
>
>
> Key: HIVE-16898
> URL: https://issues.apache.org/jira/browse/HIVE-16898
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: Daniel Dai
> Fix For: 3.0.0
>
> Attachments: HIVE-16898.1.patch, HIVE-16898.2.patch,
> HIVE-16898.3.patch, HIVE-16898.4.patch, HIVE-16898.5.patch,
> HIVE-16898.6.patch, HIVE-16898.7.patch
>
>
> time between deciding the source and destination path for distcp to invoking
> of distcp can have a change of the source file, hence distcp might copy the
> wrong file to destination, hence we should an additional check on the
> checksum of the source file path after distcp finishes to make sure the path
> didnot change during the copy process. if it has take additional steps to
> delete the previous file on destination and copy the new source and repeat
> the same pr
[jira] [Commented] (HIVE-17371) Move tokenstores to metastore module
[
https://issues.apache.org/jira/browse/HIVE-17371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175834#comment-16175834
]
Hive QA commented on HIVE-17371:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888327/HIVE-17371.01.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11051 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6933/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6933/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6933/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888327 - PreCommit-HIVE-Build
> Move tokenstores to metastore module
>
>
> Key: HIVE-17371
> URL: https://issues.apache.org/jira/browse/HIVE-17371
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17371.01.patch
>
>
> The {{getTokenStore}} method will not work for the {{DBTokenStore}} and
> {{ZKTokenStore}} since they implement
> {{org.apache.hadoop.hive.thrift.DelegationTokenStore}} instead of
> {{org.apache.hadoop.hive.metastore.security.DelegationTokenStore}}
> {code}
> private DelegationTokenStore getTokenStore(Configuration conf) throws
> IOException {
> String tokenStoreClassName =
> MetastoreConf.getVar(conf,
> MetastoreConf.ConfVars.DELEGATION_TOKEN_STORE_CLS, "");
> // The second half of this if is to catch cases where users are passing
> in a HiveConf for
> // configuration. It will have set the default value of
> // "hive.cluster.delegation.token.store .class" to
> // "org.apache.hadoop.hive.thrift.MemoryTokenStore" as part of its
> construction. But this is
> // the hive-shims version of the memory store. We want to convert this
> to our default value.
> if (StringUtils.isBlank(tokenStoreClassName) ||
>
> "org.apache.hadoop.hive.thrift.MemoryTokenStore".equals(tokenStoreClassName))
> {
> return new MemoryTokenStore();
> }
> try {
> Class storeClass =
>
> Class.forName(tokenStoreClassName).asSubclass(DelegationTokenStore.class);
> return ReflectionUtils.newInstance(storeClass, conf);
> } catch (ClassNotFoundException e) {
> throw new IOException("Error initializing delegation token store: " +
> tokenStoreClassName, e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17565) NullPointerException occurs when hive.optimize.skewjoin and hive.auto.convert.join are switched on at the same time
[ https://issues.apache.org/jira/browse/HIVE-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175846#comment-16175846 ] liyunzhang_intel commented on HIVE-17565: - i can reproduce it in Hive on MR in commit(fafa953), will investigate it later. > NullPointerException occurs when hive.optimize.skewjoin and > hive.auto.convert.join are switched on at the same time > --- > > Key: HIVE-17565 > URL: https://issues.apache.org/jira/browse/HIVE-17565 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Xin Hao >Assignee: liyunzhang_intel > > (A)NullPointerException occurs when hive.optimize.skewjoin and > hive.auto.convert.join are switched on at the same time. > Could pass when hive.optimize.skewjoin=true and hive.auto.convert.join=false. > (B)Hive Version: > Found on Apache Hive 1.2.1 > (C)Workload: > (1)TPCx-BB Q19 > (2) A small case as below，which was actually simplified from Q19： > SELECT * > FROM store_returns sr, > ( > SELECT d1.d_date_sk > FROM date_dim d1, date_dim d2 > WHERE d1.d_week_seq = d2.d_week_seq > ) sr_dateFilter > WHERE sr.sr_returned_date_sk = d_date_sk; > (D)Exception Error Message: > Error: java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:194) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:223) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1051) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055) > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:490) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170) > ... 8 more -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17538) Enhance estimation of stats to estimate even if only one column is missing stats
[ https://issues.apache.org/jira/browse/HIVE-17538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17538: --- Status: Open (was: Patch Available) > Enhance estimation of stats to estimate even if only one column is missing > stats > > > Key: HIVE-17538 > URL: https://issues.apache.org/jira/browse/HIVE-17538 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17538.1.patch, HIVE-17538.2.patch > > > HIVE-16811 provided support for estimating statistics in absence of stats. > But that estimation is done if and only if statistics are missing for all > columns. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17538) Enhance estimation of stats to estimate even if only one column is missing stats
[ https://issues.apache.org/jira/browse/HIVE-17538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17538: --- Status: Patch Available (was: Open) Updated the logic in latest patch(2). Running it to get failures. [~ashutoshc] I'll update the code to use sets in next patch. > Enhance estimation of stats to estimate even if only one column is missing > stats > > > Key: HIVE-17538 > URL: https://issues.apache.org/jira/browse/HIVE-17538 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17538.1.patch, HIVE-17538.2.patch > > > HIVE-16811 provided support for estimating statistics in absence of stats. > But that estimation is done if and only if statistics are missing for all > columns. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17538) Enhance estimation of stats to estimate even if only one column is missing stats
[ https://issues.apache.org/jira/browse/HIVE-17538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17538: --- Attachment: HIVE-17538.2.patch > Enhance estimation of stats to estimate even if only one column is missing > stats > > > Key: HIVE-17538 > URL: https://issues.apache.org/jira/browse/HIVE-17538 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17538.1.patch, HIVE-17538.2.patch > > > HIVE-16811 provided support for estimating statistics in absence of stats. > But that estimation is done if and only if statistics are missing for all > columns. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17573) LLAP: JDK9 support fixes
[ https://issues.apache.org/jira/browse/HIVE-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-17573: -- Assignee: Gopal V > LLAP: JDK9 support fixes > > > Key: HIVE-17573 > URL: https://issues.apache.org/jira/browse/HIVE-17573 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Gopal V > > The perf diff between JDK8 -> JDK9 seems to be significant. > TPC-H Q6 on JDK8 takes 32s on a single node + 1 Tb scale warehouse. > TPC-H Q6 on JDK9 takes 19s on the same host + same data. > The performance difference seems to come from better JIT and better NUMA > handling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated HIVE-17139: -- Attachment: HIVE-17139.13.patch > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.10.patch, HIVE-17139.11.patch, > HIVE-17139.12.patch, HIVE-17139.13.patch, HIVE-17139.1.patch, > HIVE-17139.2.patch, HIVE-17139.3.patch, HIVE-17139.4.patch, > HIVE-17139.5.patch, HIVE-17139.6.patch, HIVE-17139.7.patch, > HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17536) StatsUtil::getBasicStatForTable doesn't distinguish b/w absence of statistics or zero stats
[
https://issues.apache.org/jira/browse/HIVE-17536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175909#comment-16175909
]
Hive QA commented on HIVE-17536:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888369/HIVE-17536.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11051 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=240)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout
(batchId=228)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6935/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6935/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6935/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12888369 - PreCommit-HIVE-Build
> StatsUtil::getBasicStatForTable doesn't distinguish b/w absence of statistics
> or zero stats
> ---
>
> Key: HIVE-17536
> URL: https://issues.apache.org/jira/browse/HIVE-17536
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-17536.1.patch, HIVE-17536.2.patch,
> HIVE-17536.3.patch, HIVE-17536.4.patch, HIVE-17536.5.patch, HIVE-17536.6.patch
>
>
> This method returns zero for both of the following cases:
> * Statistics are missing in metastore
> * Actual stats e.g. number of rows are zero
> It'll be good for this method to return e.g. -1 in absence of statistics
> instead of assuming it to be zero.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17426) Execution framework in hive to run tasks in parallel
[
https://issues.apache.org/jira/browse/HIVE-17426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175951#comment-16175951
]
Thejas M Nair commented on HIVE-17426:
--
[~anishek] Looks like this needs a rebase. Can you please take a look ?
> Execution framework in hive to run tasks in parallel
>
>
> Key: HIVE-17426
> URL: https://issues.apache.org/jira/browse/HIVE-17426
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 3.0.0
>
> Attachments: HIVE-17426.0.patch, HIVE-17426.1.patch,
> HIVE-17426.2.patch, HIVE-17426.3.patch, HIVE-17426.4.patch, HIVE-17426.5.patch
>
>
> the execution framework currently only runs MR / Spark Tasks in parallel
> when {{set hive.exec.parallel=true}}.
> Allow other types of tasks to run in parallel as well to support replication
> scenarios in hive.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17560) HiveMetastore doesn't start in secure cluster if repl change manager is enabled
[ https://issues.apache.org/jira/browse/HIVE-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-17560: - Resolution: Fixed Status: Resolved (was: Patch Available) > HiveMetastore doesn't start in secure cluster if repl change manager is > enabled > --- > > Key: HIVE-17560 > URL: https://issues.apache.org/jira/browse/HIVE-17560 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 3.0.0 > > Attachments: HIVE-17560.1.patch > > > When hive.repl.cm.enabled=true, ReplChangeManager tries to access HDFS before > metastore does kerberos login using keytab. > Metastore startup code doesn't do an explicit login using keytab, but instead > relies on kinit by saslserver for use by thrift to do it. > It would be cleaner to do an explicit UGI.loginFromKeytab instead to avoid > such issues in future as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17574) Avoid multiple copies of HDFS-based jars when localizing job-jars
[
https://issues.apache.org/jira/browse/HIVE-17574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175967#comment-16175967
]
Hive QA commented on HIVE-17574:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888397/HIVE-17574.1-branch-2.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 59 failed/errored test(s), 9935 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=244)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=225)
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=167)
[acid_globallimit.q,alter_merge_2_orc.q]
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=173)
[infer_bucket_sort_reducers_power_two.q,list_bucket_dml_10.q,orc_merge9.q,orc_merge6.q,leftsemijoin_mr.q,bucket6.q,bucketmapjoin7.q,uber_reduce.q,empty_dir_in_table.q,vector_outer_join3.q,index_bitmap_auto.q,vector_outer_join2.q,vector_outer_join1.q,orc_merge1.q,orc_merge_diff_fs.q,load_hdfs_file_with_space_in_the_name.q,scriptfile1_win.q,quotedid_smb.q,truncate_column_buckets.q,orc_merge3.q]
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=174)
[infer_bucket_sort_num_buckets.q,gen_udf_example_add10.q,insert_overwrite_directory2.q,orc_merge5.q,bucketmapjoin6.q,import_exported_table.q,vector_outer_join0.q,orc_merge4.q,temp_table_external.q,orc_merge_incompat1.q,root_dir_external_table.q,constprog_semijoin.q,auto_sortmerge_join_16.q,schemeAuthority.q,index_bitmap3.q,external_table_with_space_in_location_path.q,parallel_orderby.q,infer_bucket_sort_map_operators.q,bucketizedhiveinputformat.q,remote_script.q]
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=175)
[scriptfile1.q,vector_outer_join5.q,file_with_header_footer.q,bucket4.q,input16_cc.q,bucket5.q,infer_bucket_sort_merge.q,constprog_partitioner.q,orc_merge2.q,reduce_deduplicate.q,schemeAuthority2.q,load_fs2.q,orc_merge8.q,orc_merge_incompat2.q,infer_bucket_sort_bucketed_table.q,vector_outer_join4.q,disable_merge_for_bucketing.q,vector_inner_join.q,orc_merge7.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=118)
[bucketmapjoin4.q,bucket_map_join_spark4.q,union21.q,groupby2_noskew.q,timestamp_2.q,date_join1.q,mergejoins.q,smb_mapjoin_11.q,auto_sortmerge_join_3.q,mapjoin_test_outer.q,vectorization_9.q,merge2.q,groupby6_noskew.q,auto_join_without_localtask.q,multi_join_union.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=119)
[join_cond_pushdown_unqual4.q,union_remove_7.q,join13.q,join_vc.q,groupby_cube1.q,bucket_map_join_spark2.q,sample3.q,smb_mapjoin_19.q,stats16.q,union23.q,union.q,union31.q,cbo_udf_udaf.q,ptf_decimal.q,bucketmapjoin2.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=120)
[parallel_join1.q,union27.q,union12.q,groupby7_map_multi_single_reducer.q,varchar_join1.q,join7.q,join_reorder4.q,skewjoinopt2.q,bucketsortoptimize_insert_2.q,smb_mapjoin_17.q,script_env_var1.q,groupby7_map.q,groupby3.q,bucketsortoptimize_insert_8.q,union20.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=121)
[ptf_general_queries.q,auto_join_reordering_values.q,sample2.q,join1.q,decimal_join.q,mapjoin_subquery2.q,join32_lessSize.q,mapjoin1.q,order2.q,skewjoinopt18.q,union_remove_18.q,join25.q,groupby9.q,bucketsortoptimize_insert_6.q,ctas.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=122)
[groupby_map_ppr.q,nullgroup4_multi_distinct.q,join_rc.q,union14.q,smb_mapjoin_12.q,vector_cast_constant.q,union_remove_4.q,auto_join11.q,load_dyn_part7.q,udaf_collect_set.q,vectorization_12.q,groupby_sort_skew_1.q,groupby_sort_skew_1_23.q,smb_mapjoin_25.q,skewjoinopt12.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=123)
[skewjoinopt15.q,auto_join18.q,list_bucket_dml_2.q,input1_limit.q,load_dyn_part3.q,union_remove_14.q,auto_sortmerge_join_14.q,auto_sortmerge_join_15.q,union10.q,bucket_map_join_tez2.q,groupby5_map_skew.q,join_reorder.q,sample1.q,bucketmapjoin8.q,union34.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=124)
[avro_joins.q,skewjoinopt16.q,auto_join14.q,vectorization_14.q,auto_join26.q,stats1.q,cbo_stats.q,auto_sortmerge_join_6.q,union22.q,union_remove_24.q,union_view.q,smb_mapjoin_22.q,stats15.q,ptf_matchpath.q,transform_ppr1.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=125)
[limit_pushdown2.q,skewjoin_no
[jira] [Updated] (HIVE-17570) Fix view deletion related test failures (create_view.q etc)
[ https://issues.apache.org/jira/browse/HIVE-17570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-17570: - Resolution: Fixed Status: Resolved (was: Patch Available) Committer to master. Thanks for the patch [~taoli-hwx] > Fix view deletion related test failures (create_view.q etc) > --- > > Key: HIVE-17570 > URL: https://issues.apache.org/jira/browse/HIVE-17570 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17570.1.patch, HIVE-17570.2.patch > > > Fixing the bug introduced by HIVE-17459. Sorry that did not capture that in a > timely fashion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Work started] (HIVE-17571) update sql standard authorization config whitelist to include distcp options for replication
[ https://issues.apache.org/jira/browse/HIVE-17571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-17571 started by Thejas M Nair. > update sql standard authorization config whitelist to include distcp options > for replication > > > Key: HIVE-17571 > URL: https://issues.apache.org/jira/browse/HIVE-17571 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-17571.1.patch > > > Additional distcp config options (added in HIVE-16686) need to be added to > whitelist of configs that can be updated at runtime, for sql standard > authorization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17571) update sql standard authorization config whitelist to include distcp options for replication
[ https://issues.apache.org/jira/browse/HIVE-17571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-17571: - Attachment: HIVE-17571.1.patch > update sql standard authorization config whitelist to include distcp options > for replication > > > Key: HIVE-17571 > URL: https://issues.apache.org/jira/browse/HIVE-17571 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-17571.1.patch > > > Additional distcp config options (added in HIVE-16686) need to be added to > whitelist of configs that can be updated at runtime, for sql standard > authorization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
