[jira] [Commented] (HIVE-17049) hive doesn't support chinese comments for columns
[ https://issues.apache.org/jira/browse/HIVE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076056#comment-16076056 ] ZhangBing Lin commented on HIVE-17049: -- maybe I use hive 2.1.0 > hive doesn't support chinese comments for columns > - > > Key: HIVE-17049 > URL: https://issues.apache.org/jira/browse/HIVE-17049 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 1.2.1 > Environment: hive 1.2.1 in HDP >Reporter: liugaopeng > > 1. alter table stg.test_chinese change chinesetitle chinesetitle tinyint > comment '中文'; > 2. desc stg.test_chinese; > Result: chinese comment "中文" becase "??" > also, if i modify the comment via hive view, it will still display the messy > code "??". > I did some testing, but cannot fix it, such as: > 1. change the hive.COLUMNS_V2 to UTF-8 chartset. > 2. append the characterEncoding=UTF-8 to hive_to_mysqlmetadata url > i found some ideas that need to apply some patch to fix it, but seems they > all effects in 0.x version, i use the 1.2.1 version. > Please give some guidance. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Issue Comment Deleted] (HIVE-16922) Typo in serde.thrift: COLLECTION_DELIM = "colelction.delim"
[ https://issues.apache.org/jira/browse/HIVE-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bing Li updated HIVE-16922: --- Comment: was deleted (was: The patch is based on master branch.) > Typo in serde.thrift: COLLECTION_DELIM = "colelction.delim" > --- > > Key: HIVE-16922 > URL: https://issues.apache.org/jira/browse/HIVE-16922 > Project: Hive > Issue Type: Bug > Components: Thrift API >Reporter: Dudu Markovitz >Assignee: Bing Li > Attachments: HIVE-16922.1.patch > > > https://github.com/apache/hive/blob/master/serde/if/serde.thrift > Typo in serde.thrift: > COLLECTION_DELIM = "colelction.delim" > (*colelction* instead of *collection*) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-16922) Typo in serde.thrift: COLLECTION_DELIM = "colelction.delim"
[ https://issues.apache.org/jira/browse/HIVE-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076039#comment-16076039 ] Bing Li edited comment on HIVE-16922 at 7/6/17 7:02 AM: This patch file is created based on master branch. The Thrift codes are generated by thrift-0.9.3. was (Author: libing): This patch file is created based on master branch. > Typo in serde.thrift: COLLECTION_DELIM = "colelction.delim" > --- > > Key: HIVE-16922 > URL: https://issues.apache.org/jira/browse/HIVE-16922 > Project: Hive > Issue Type: Bug > Components: Thrift API >Reporter: Dudu Markovitz >Assignee: Bing Li > Attachments: HIVE-16922.1.patch > > > https://github.com/apache/hive/blob/master/serde/if/serde.thrift > Typo in serde.thrift: > COLLECTION_DELIM = "colelction.delim" > (*colelction* instead of *collection*) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17049) hive doesn't support chinese comments for columns
[ https://issues.apache.org/jira/browse/HIVE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076053#comment-16076053 ] ZhangBing Lin edited comment on HIVE-17049 at 7/6/17 7:09 AM: -- [~liugaopeng],what's your qq number?I add was (Author: linzhangbing): [~liugaopeng],your qq number > hive doesn't support chinese comments for columns > - > > Key: HIVE-17049 > URL: https://issues.apache.org/jira/browse/HIVE-17049 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 1.2.1 > Environment: hive 1.2.1 in HDP >Reporter: liugaopeng > > 1. alter table stg.test_chinese change chinesetitle chinesetitle tinyint > comment '中文'; > 2. desc stg.test_chinese; > Result: chinese comment "中文" becase "??" > also, if i modify the comment via hive view, it will still display the messy > code "??". > I did some testing, but cannot fix it, such as: > 1. change the hive.COLUMNS_V2 to UTF-8 chartset. > 2. append the characterEncoding=UTF-8 to hive_to_mysqlmetadata url > i found some ideas that need to apply some patch to fix it, but seems they > all effects in 0.x version, i use the 1.2.1 version. > Please give some guidance. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17049) hive doesn't support chinese comments for columns
[ https://issues.apache.org/jira/browse/HIVE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076053#comment-16076053 ] ZhangBing Lin edited comment on HIVE-17049 at 7/6/17 7:09 AM: -- [~liugaopeng],your qq number was (Author: linzhangbing): qq:1018917998 > hive doesn't support chinese comments for columns > - > > Key: HIVE-17049 > URL: https://issues.apache.org/jira/browse/HIVE-17049 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 1.2.1 > Environment: hive 1.2.1 in HDP >Reporter: liugaopeng > > 1. alter table stg.test_chinese change chinesetitle chinesetitle tinyint > comment '中文'; > 2. desc stg.test_chinese; > Result: chinese comment "中文" becase "??" > also, if i modify the comment via hive view, it will still display the messy > code "??". > I did some testing, but cannot fix it, such as: > 1. change the hive.COLUMNS_V2 to UTF-8 chartset. > 2. append the characterEncoding=UTF-8 to hive_to_mysqlmetadata url > i found some ideas that need to apply some patch to fix it, but seems they > all effects in 0.x version, i use the 1.2.1 version. > Please give some guidance. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17018) Small table is converted to map join even the total size of small tables exceeds the threshold(hive.auto.convert.join.noconditionaltask.size)
[
https://issues.apache.org/jira/browse/HIVE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076105#comment-16076105
]
liyunzhang_intel commented on HIVE-17018:
-
[~csun]: attached an example(HIVE-17018_data_init.q, HIVE-17018.q)
HIVE-17018
{code}
set hive.auto.convert.join.noconditionaltask.size=460;
explain select * from src,t1,t2,t3 where src.key=t1.key1 and src.value=t2.key2
and t1.value1=t3.value3;
{code}
the explain
{code}
STAGE DEPENDENCIES:
Stage-2 is a root stage
Stage-3 depends on stages: Stage-2
Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
Stage: Stage-2
Spark
DagName: root_20170706020353_8bc59675-8374-4ec7-ab9c-4904cd5fcadb:2
Vertices:
Map 5
Map Operator Tree:
TableScan
alias: t3
Statistics: Num rows: 2 Data size: 460 Basic stats: COMPLETE
Column stats: NONE
Filter Operator
predicate: value3 is not null (type: boolean)
Statistics: Num rows: 2 Data size: 460 Basic stats:
COMPLETE Column stats: NONE
Spark HashTable Sink Operator
keys:
0 _col6 (type: string)
1 value3 (type: string)
Local Work:
Map Reduce Local Work
Stage: Stage-3
Spark
DagName: root_20170706020353_8bc59675-8374-4ec7-ab9c-4904cd5fcadb:3
Vertices:
Map 3
Map Operator Tree:
TableScan
alias: t1
Statistics: Num rows: 2 Data size: 460 Basic stats: COMPLETE
Column stats: NONE
Filter Operator
predicate: (key1 is not null and value1 is not null) (type:
boolean)
Statistics: Num rows: 2 Data size: 460 Basic stats:
COMPLETE Column stats: NONE
Spark HashTable Sink Operator
keys:
0 key (type: string)
1 key1 (type: string)
Local Work:
Map Reduce Local Work
Stage: Stage-1
Spark
Edges:
Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 12), Map 4 (PARTITION-LEVEL
SORT, 12)
DagName: root_20170706020353_8bc59675-8374-4ec7-ab9c-4904cd5fcadb:1
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: src
Statistics: Num rows: 29 Data size: 5812 Basic stats:
COMPLETE Column stats: NONE
Filter Operator
predicate: (key is not null and value is not null) (type:
boolean)
Statistics: Num rows: 29 Data size: 5812 Basic stats:
COMPLETE Column stats: NONE
Map Join Operator
condition map:
Inner Join 0 to 1
keys:
0 key (type: string)
1 key1 (type: string)
outputColumnNames: _col0, _col1, _col5, _col6
input vertices:
1 Map 3
Statistics: Num rows: 31 Data size: 6393 Basic stats:
COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: string)
sort order: +
Map-reduce partition columns: _col1 (type: string)
Statistics: Num rows: 31 Data size: 6393 Basic stats:
COMPLETE Column stats: NONE
value expressions: _col0 (type: string), _col5 (type:
string), _col6 (type: string)
Local Work:
Map Reduce Local Work
Map 4
Map Operator Tree:
TableScan
alias: t2
Statistics: Num rows: 2 Data size: 460 Basic stats: COMPLETE
Column stats: NONE
Filter Operator
predicate: key2 is not null (type: boolean)
Statistics: Num rows: 2 Data size: 460 Basic stats:
COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: key2 (type: string)
sort order: +
Map-reduce partition columns: key2 (type: string)
Statistics: Num rows: 2 Data size: 460 Basic stats:
COMPLETE Column stats: NONE
value expressions: value2 (type: string)
Reducer 2
Local Work:
Map Reduce Local Work
Reduce Operator Tree:
Join Operator
condition map:
Inner Join 0 to 1
keys:
0 _col1 (type: string)
1 key2 (type: string)
outputColumnNames: _col0, _col1, _col5, _col
[jira] [Updated] (HIVE-17010) Fix the overflow problem of Long type in SetSparkReducerParallelism
[
https://issues.apache.org/jira/browse/HIVE-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang_intel updated HIVE-17010:
Attachment: HIVE-17010.2.patch
[~csun]: update HIVE-17010.2.patch to use {{StatsUtils.safeAdd}} to solve the
overflow. help review
> Fix the overflow problem of Long type in SetSparkReducerParallelism
> ---
>
> Key: HIVE-17010
> URL: https://issues.apache.org/jira/browse/HIVE-17010
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-17010.1.patch, HIVE-17010.2.patch
>
>
> We use
> [numberOfByteshttps://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L129]
> to collect the numberOfBytes of sibling of specified RS. We use Long type
> and it happens overflow when the data is too big. After happening this
> situation, the parallelism is decided by
> [sparkMemoryAndCores.getSecond()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L184]
> if spark.dynamic.allocation.enabled is true, sparkMemoryAndCores.getSecond
> is a dymamic value which is decided by spark runtime. For example, the value
> of sparkMemoryAndCores.getSecond is 5 or 15 randomly. There is possibility
> that the value may be 1. The may problem here is the overflow of addition of
> Long type. You can reproduce the overflow problem by following code
> {code}
> public static void main(String[] args) {
> long a1= 9223372036854775807L;
> long a2=1022672;
> long res = a1+a2;
> System.out.println(res); //-9223372036853753137
> BigInteger b1= BigInteger.valueOf(a1);
> BigInteger b2 = BigInteger.valueOf(a2);
> BigInteger bigRes = b1.add(b2);
> System.out.println(bigRes); //9223372036855798479
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17010) Fix the overflow problem of Long type in SetSparkReducerParallelism
[
https://issues.apache.org/jira/browse/HIVE-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang_intel updated HIVE-17010:
Status: Patch Available (was: Open)
> Fix the overflow problem of Long type in SetSparkReducerParallelism
> ---
>
> Key: HIVE-17010
> URL: https://issues.apache.org/jira/browse/HIVE-17010
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-17010.1.patch, HIVE-17010.2.patch
>
>
> We use
> [numberOfByteshttps://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L129]
> to collect the numberOfBytes of sibling of specified RS. We use Long type
> and it happens overflow when the data is too big. After happening this
> situation, the parallelism is decided by
> [sparkMemoryAndCores.getSecond()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L184]
> if spark.dynamic.allocation.enabled is true, sparkMemoryAndCores.getSecond
> is a dymamic value which is decided by spark runtime. For example, the value
> of sparkMemoryAndCores.getSecond is 5 or 15 randomly. There is possibility
> that the value may be 1. The may problem here is the overflow of addition of
> Long type. You can reproduce the overflow problem by following code
> {code}
> public static void main(String[] args) {
> long a1= 9223372036854775807L;
> long a2=1022672;
> long res = a1+a2;
> System.out.println(res); //-9223372036853753137
> BigInteger b1= BigInteger.valueOf(a1);
> BigInteger b2 = BigInteger.valueOf(a2);
> BigInteger bigRes = b1.add(b2);
> System.out.println(bigRes); //9223372036855798479
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16893) move replication dump related work in semantic analysis phase to execution phase using a task
[ https://issues.apache.org/jira/browse/HIVE-16893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076130#comment-16076130 ] anishek commented on HIVE-16893: Thanks [~daijy]for the commit and [~sankarh] for the review. > move replication dump related work in semantic analysis phase to execution > phase using a task > - > > Key: HIVE-16893 > URL: https://issues.apache.org/jira/browse/HIVE-16893 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek > Fix For: 3.0.0 > > Attachments: HIVE-16893.2.patch, HIVE-16893.3.patch, > HIVE-16893.4.patch > > > Since we run in to the possibility of creating a large number tasks during > replication bootstrap dump > * we may not be able to hold all of them in memory for really large > databases, which might not hold true once we complete HIVE-16892 > * Also a compile time lock is taken such that only one query is run in this > phase which in replication bootstrap scenario is going to be a very long > running task and hence moving it to execution phase will limit the lock > period in compile phase. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16922) Typo in serde.thrift: COLLECTION_DELIM = "colelction.delim"
[
https://issues.apache.org/jira/browse/HIVE-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076135#comment-16076135
]
Hive QA commented on HIVE-16922:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12875864/HIVE-16922.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10832 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_create]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=232)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=177)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5905/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5905/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5905/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12875864 - PreCommit-HIVE-Build
> Typo in serde.thrift: COLLECTION_DELIM = "colelction.delim"
> ---
>
> Key: HIVE-16922
> URL: https://issues.apache.org/jira/browse/HIVE-16922
> Project: Hive
> Issue Type: Bug
> Components: Thrift API
>Reporter: Dudu Markovitz
>Assignee: Bing Li
> Attachments: HIVE-16922.1.patch
>
>
> https://github.com/apache/hive/blob/master/serde/if/serde.thrift
> Typo in serde.thrift:
> COLLECTION_DELIM = "colelction.delim"
> (*colelction* instead of *collection*)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-14826) Support vectorization for Parquet
[ https://issues.apache.org/jira/browse/HIVE-14826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-14826: Attachment: Hive_Parquet_vectorization_design_document.pdf > Support vectorization for Parquet > - > > Key: HIVE-14826 > URL: https://issues.apache.org/jira/browse/HIVE-14826 > Project: Hive > Issue Type: New Feature >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: Hive_Parquet_vectorization_design_document.pdf > > > Parquet vectorized reader can improve both throughput and also leverages > existing Hive vectorization execution engine. This is an umbrella ticket to > track this feature. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17001) Insert overwrite table doesn't clean partition directory on HDFS if partition is missing from HMS
[
https://issues.apache.org/jira/browse/HIVE-17001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara updated HIVE-17001:
---
Description:
Insert overwrite table should clear existing data before creating the new data
files.
For a partitioned table we will clean any folder of existing partitions on
HDFS, however if the partition folder exists only on HDFS and the partition
definition is missing in HMS, the folder is not cleared.
Reproduction steps:
1. CREATE TABLE test( col1 string) PARTITIONED BY (ds string);
2. INSERT INTO test PARTITION(ds='p1') values ('a');
3. Copy the data to a different folder with different name.
4. ALTER TABLE test DROP PARTITION (ds='p1');
5. Recreate the partition directory, copy and rename the data file back
6. INSERT OVERWRITE TABLE test PARTITION(ds='p1') values ('b');
7. SELECT * from test;
will result in 2 records being returned instead of 1.
was:
Insert overwrite table should clear existing data before creating the new data
files.
For a partitioned table we will clean any folder of existing partitions on
HDFS, however if the partition folder exists only on HDFS and the partition
definition is missing in HMS, the folder is not cleared.
Reproduction steps:
1. CREATE TABLE test( col1 string) PARTITIONED BY (ds string);
2. INSERT INTO test PARTITION(ds='p1') values ('a');
3. Copy the data to a different folder with different name.
4. ALTER TABLE test DROP PARTITION (ds='p1');
5. Recreate the partition directory, copy and rename the data file back
6. INSERT INTO test PARTITION(ds='p1') values ('b');
7. SELECT * from test;
will result in 2 records being returned instead of 1.
> Insert overwrite table doesn't clean partition directory on HDFS if partition
> is missing from HMS
> -
>
> Key: HIVE-17001
> URL: https://issues.apache.org/jira/browse/HIVE-17001
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Metastore
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-17001.01.patch
>
>
> Insert overwrite table should clear existing data before creating the new
> data files.
> For a partitioned table we will clean any folder of existing partitions on
> HDFS, however if the partition folder exists only on HDFS and the partition
> definition is missing in HMS, the folder is not cleared.
> Reproduction steps:
> 1. CREATE TABLE test( col1 string) PARTITIONED BY (ds string);
> 2. INSERT INTO test PARTITION(ds='p1') values ('a');
> 3. Copy the data to a different folder with different name.
> 4. ALTER TABLE test DROP PARTITION (ds='p1');
> 5. Recreate the partition directory, copy and rename the data file back
> 6. INSERT OVERWRITE TABLE test PARTITION(ds='p1') values ('b');
> 7. SELECT * from test;
> will result in 2 records being returned instead of 1.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17010) Fix the overflow problem of Long type in SetSparkReducerParallelism
[
https://issues.apache.org/jira/browse/HIVE-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076200#comment-16076200
]
Hive QA commented on HIVE-17010:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12875874/HIVE-17010.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10832 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_reducers_power_two]
(batchId=167)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=232)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=177)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation (batchId=226)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5906/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5906/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5906/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12875874 - PreCommit-HIVE-Build
> Fix the overflow problem of Long type in SetSparkReducerParallelism
> ---
>
> Key: HIVE-17010
> URL: https://issues.apache.org/jira/browse/HIVE-17010
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-17010.1.patch, HIVE-17010.2.patch
>
>
> We use
> [numberOfByteshttps://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L129]
> to collect the numberOfBytes of sibling of specified RS. We use Long type
> and it happens overflow when the data is too big. After happening this
> situation, the parallelism is decided by
> [sparkMemoryAndCores.getSecond()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L184]
> if spark.dynamic.allocation.enabled is true, sparkMemoryAndCores.getSecond
> is a dymamic value which is decided by spark runtime. For example, the value
> of sparkMemoryAndCores.getSecond is 5 or 15 randomly. There is possibility
> that the value may be 1. The may problem here is the overflow of addition of
> Long type. You can reproduce the overflow problem by following code
> {code}
> public static void main(String[] args) {
> long a1= 9223372036854775807L;
> long a2=1022672;
> long res = a1+a2;
> System.out.println(res); //-9223372036853753137
> BigInteger b1= BigInteger.valueOf(a1);
> BigInteger b2 = BigInteger.valueOf(a2);
> BigInteger bigRes = b1.add(b2);
> System.out.println(bigRes); //9223372036855798479
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17001) Insert overwrite table doesn't clean partition directory on HDFS if partition is missing from HMS
[
https://issues.apache.org/jira/browse/HIVE-17001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076209#comment-16076209
]
Barna Zsombor Klara commented on HIVE-17001:
[~spena] No, the file is not moved outside in the test just renamed from
00_0 to 00_1, it is there in the same partition directory (unless I
made a typo I'm not seeing right now).
[~ngangam] Yes I made a mistake in the description, the insert should be an
insert overwrite table, let me correct it. But the behaviour is the same, the
datafile is not overwritten with insert overwrite either.
> Insert overwrite table doesn't clean partition directory on HDFS if partition
> is missing from HMS
> -
>
> Key: HIVE-17001
> URL: https://issues.apache.org/jira/browse/HIVE-17001
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Metastore
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-17001.01.patch
>
>
> Insert overwrite table should clear existing data before creating the new
> data files.
> For a partitioned table we will clean any folder of existing partitions on
> HDFS, however if the partition folder exists only on HDFS and the partition
> definition is missing in HMS, the folder is not cleared.
> Reproduction steps:
> 1. CREATE TABLE test( col1 string) PARTITIONED BY (ds string);
> 2. INSERT INTO test PARTITION(ds='p1') values ('a');
> 3. Copy the data to a different folder with different name.
> 4. ALTER TABLE test DROP PARTITION (ds='p1');
> 5. Recreate the partition directory, copy and rename the data file back
> 6. INSERT OVERWRITE TABLE test PARTITION(ds='p1') values ('b');
> 7. SELECT * from test;
> will result in 2 records being returned instead of 1.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17047) Allow table property to be populated to jobConf to make FixedLengthInputFormat work
[
https://issues.apache.org/jira/browse/HIVE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-17047:
Description:
underlined textTo make FixedLengthInputFormat work in Hive, we need table
specific value for the configuration "fixedlengthinputformat.record.length".
Right now the best place would be table property. Unfortunately, table property
is not alway populated to InputFormat configurations because of this in
HiveInputFormat:
{code}
PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
if ((part != null) && (part.getTableDesc() != null)) {
{code}
was:
To make FixedLengthInputFormat work in Hive, we need table specific value for
the configuration "fixedlengthinputformat.record.length". Right now the best
place would be table property. Unfortunately, table property is not alway
populated to InputFormat configurations because of this in HiveInputFormat:
{code}
PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
if ((part != null) && (part.getTableDesc() != null)) {
{code}
> Allow table property to be populated to jobConf to make
> FixedLengthInputFormat work
> ---
>
> Key: HIVE-17047
> URL: https://issues.apache.org/jira/browse/HIVE-17047
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-17047.1.patch
>
>
> underlined textTo make FixedLengthInputFormat work in Hive, we need table
> specific value for the configuration "fixedlengthinputformat.record.length".
> Right now the best place would be table property. Unfortunately, table
> property is not alway populated to InputFormat configurations because of this
> in HiveInputFormat:
> {code}
> PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
> if ((part != null) && (part.getTableDesc() != null)) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17047) Allow table property to be populated to jobConf to make FixedLengthInputFormat work
[
https://issues.apache.org/jira/browse/HIVE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-17047:
Description:
To make FixedLengthInputFormat work in Hive, we need table specific value for
the configuration "fixedlengthinputformat.record.length". Right now the best
place would be table property. Unfortunately, table property is not alway
populated to InputFormat configurations because of this in HiveInputFormat:
{code}
PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
if ((part != null) && (part.getTableDesc() != null)) {
{code}
was:
underlined textTo make FixedLengthInputFormat work in Hive, we need table
specific value for the configuration "fixedlengthinputformat.record.length".
Right now the best place would be table property. Unfortunately, table property
is not alway populated to InputFormat configurations because of this in
HiveInputFormat:
{code}
PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
if ((part != null) && (part.getTableDesc() != null)) {
{code}
> Allow table property to be populated to jobConf to make
> FixedLengthInputFormat work
> ---
>
> Key: HIVE-17047
> URL: https://issues.apache.org/jira/browse/HIVE-17047
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-17047.1.patch
>
>
> To make FixedLengthInputFormat work in Hive, we need table specific value for
> the configuration "fixedlengthinputformat.record.length". Right now the best
> place would be table property. Unfortunately, table property is not alway
> populated to InputFormat configurations because of this in HiveInputFormat:
> {code}
> PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
> if ((part != null) && (part.getTableDesc() != null)) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17047) Allow table property to be populated to jobConf to make FixedLengthInputFormat work
[
https://issues.apache.org/jira/browse/HIVE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich reassigned HIVE-17047:
---
Assignee: Zoltan Haindrich (was: Zhiyuan Yang)
> Allow table property to be populated to jobConf to make
> FixedLengthInputFormat work
> ---
>
> Key: HIVE-17047
> URL: https://issues.apache.org/jira/browse/HIVE-17047
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zoltan Haindrich
> Attachments: HIVE-17047.1.patch
>
>
> To make FixedLengthInputFormat work in Hive, we need table specific value for
> the configuration "fixedlengthinputformat.record.length". Right now the best
> place would be table property. Unfortunately, table property is not alway
> populated to InputFormat configurations because of this in HiveInputFormat:
> {code}
> PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
> if ((part != null) && (part.getTableDesc() != null)) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17047) Allow table property to be populated to jobConf to make FixedLengthInputFormat work
[
https://issues.apache.org/jira/browse/HIVE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-17047:
Attachment: HIVE-17047-branch-1.1.patch
> Allow table property to be populated to jobConf to make
> FixedLengthInputFormat work
> ---
>
> Key: HIVE-17047
> URL: https://issues.apache.org/jira/browse/HIVE-17047
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zoltan Haindrich
> Attachments: HIVE-17047.1.patch, HIVE-17047-branch-1.1.patch
>
>
> To make FixedLengthInputFormat work in Hive, we need table specific value for
> the configuration "fixedlengthinputformat.record.length". Right now the best
> place would be table property. Unfortunately, table property is not alway
> populated to InputFormat configurations because of this in HiveInputFormat:
> {code}
> PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
> if ((part != null) && (part.getTableDesc() != null)) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17047) Allow table property to be populated to jobConf to make FixedLengthInputFormat work
[
https://issues.apache.org/jira/browse/HIVE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich reassigned HIVE-17047:
---
Assignee: Zhiyuan Yang (was: Zoltan Haindrich)
> Allow table property to be populated to jobConf to make
> FixedLengthInputFormat work
> ---
>
> Key: HIVE-17047
> URL: https://issues.apache.org/jira/browse/HIVE-17047
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-17047.1.patch, HIVE-17047-branch-1.1.patch
>
>
> To make FixedLengthInputFormat work in Hive, we need table specific value for
> the configuration "fixedlengthinputformat.record.length". Right now the best
> place would be table property. Unfortunately, table property is not alway
> populated to InputFormat configurations because of this in HiveInputFormat:
> {code}
> PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
> if ((part != null) && (part.getTableDesc() != null)) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17014) Password File Encryption for HiveServer2 Client
[ https://issues.apache.org/jira/browse/HIVE-17014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Gudikov updated HIVE-17014: Summary: Password File Encryption for HiveServer2 Client (was: Password File Encryption) > Password File Encryption for HiveServer2 Client > --- > > Key: HIVE-17014 > URL: https://issues.apache.org/jira/browse/HIVE-17014 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vlad Gudikov > > The main point of this file is to encrypt password file that is used for > beeline connection using -w key. Any ideas or proposals would be great. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17047) Allow table property to be populated to jobConf to make FixedLengthInputFormat work
[
https://issues.apache.org/jira/browse/HIVE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076295#comment-16076295
]
Zoltan Haindrich commented on HIVE-17047:
-
[~aplusplus] I've uploaded your patch to run against branch-1.
+1 the change looks good to me; it seems to me that
{{HiveInputFormat.getPartitionDescFromPath}} might be also
problematic...possibly it didnt cause any trouble...
> Allow table property to be populated to jobConf to make
> FixedLengthInputFormat work
> ---
>
> Key: HIVE-17047
> URL: https://issues.apache.org/jira/browse/HIVE-17047
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-17047.1.patch, HIVE-17047-branch-1.1.patch
>
>
> To make FixedLengthInputFormat work in Hive, we need table specific value for
> the configuration "fixedlengthinputformat.record.length". Right now the best
> place would be table property. Unfortunately, table property is not alway
> populated to InputFormat configurations because of this in HiveInputFormat:
> {code}
> PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
> if ((part != null) && (part.getTableDesc() != null)) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17014) Password File Encryption for HiveServer2 Client
[ https://issues.apache.org/jira/browse/HIVE-17014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-17014: Fix Version/s: 2.1.2 > Password File Encryption for HiveServer2 Client > --- > > Key: HIVE-17014 > URL: https://issues.apache.org/jira/browse/HIVE-17014 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vlad Gudikov > Fix For: 2.1.2 > > > The main point of this file is to encrypt password file that is used for > beeline connection using -w key. Any ideas or proposals would be great. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17014) Password File Encryption for HiveServer2 Client
[ https://issues.apache.org/jira/browse/HIVE-17014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-17014: --- Assignee: Vlad Gudikov > Password File Encryption for HiveServer2 Client > --- > > Key: HIVE-17014 > URL: https://issues.apache.org/jira/browse/HIVE-17014 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vlad Gudikov >Assignee: Vlad Gudikov > Fix For: 2.1.2 > > > The main point of this file is to encrypt password file that is used for > beeline connection using -w key. Any ideas or proposals would be great. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17050) Multiline queries that have comment in middle fail when executed via "beeline -e"
[
https://issues.apache.org/jira/browse/HIVE-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated HIVE-17050:
--
Attachment: HIVE-17050.1.patch
Submit a patch
> Multiline queries that have comment in middle fail when executed via "beeline
> -e"
> -
>
> Key: HIVE-17050
> URL: https://issues.apache.org/jira/browse/HIVE-17050
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
> Attachments: HIVE-17050.1.patch
>
>
> After applying HIVE-13864, multiple line queries that have comment at the end
> of one of the middle lines fail when executed via beeline -e
> {noformat}
> $ beeline -u "" -e "select 1, --test
> > 2"
> scan complete in 3ms
> ..
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Error: Error while compiling statement: FAILED: ParseException line 1:9
> cannot recognize input near '' '' '' in selection target
> (state=42000,code=4)
> Closing: 0:
> jdbc:hive2://host-10-17-80-194.coe.cloudera.com:1/default;principal=hive/host-10-17-80-194.coe.cloudera@yshi.com;ssl=true;sslTrustStore=/certs/hive/hive-keystore.jks;trustStorePassword=cloudera
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17050) Multiline queries that have comment in middle fail when executed via "beeline -e"
[
https://issues.apache.org/jira/browse/HIVE-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated HIVE-17050:
--
Assignee: Yibing Shi
Status: Patch Available (was: Open)
> Multiline queries that have comment in middle fail when executed via "beeline
> -e"
> -
>
> Key: HIVE-17050
> URL: https://issues.apache.org/jira/browse/HIVE-17050
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-17050.1.patch
>
>
> After applying HIVE-13864, multiple line queries that have comment at the end
> of one of the middle lines fail when executed via beeline -e
> {noformat}
> $ beeline -u "" -e "select 1, --test
> > 2"
> scan complete in 3ms
> ..
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Error: Error while compiling statement: FAILED: ParseException line 1:9
> cannot recognize input near '' '' '' in selection target
> (state=42000,code=4)
> Closing: 0:
> jdbc:hive2://host-10-17-80-194.coe.cloudera.com:1/default;principal=hive/host-10-17-80-194.coe.cloudera@yshi.com;ssl=true;sslTrustStore=/certs/hive/hive-keystore.jks;trustStorePassword=cloudera
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17051) Each table metadata is requested twice during query compile
[
https://issues.apache.org/jira/browse/HIVE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu reassigned HIVE-17051:
---
> Each table metadata is requested twice during query compile
> ---
>
> Key: HIVE-17051
> URL: https://issues.apache.org/jira/browse/HIVE-17051
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Labels: performance
>
> As far as I can tell, for each table referenced in a query the metadata is
> retrieved twice during compilation:
> first call:
> {noformat}
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1320)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1275)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getTableObjectByName(SemanticAnalyzer.java:10943)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1992)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1942)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:11178)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11309)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:295)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:261)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:566)
> {noformat}
> second call:
> {noformat}
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1320)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1275)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getTableObjectByName(SemanticAnalyzer.java:10943)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1992)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1942)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1934)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:431)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11320)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:295)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:261)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:566)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17050) Multiline queries that have comment in middle fail when executed via "beeline -e"
[
https://issues.apache.org/jira/browse/HIVE-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076391#comment-16076391
]
Hive QA commented on HIVE-17050:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12875902/HIVE-17050.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10832 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=237)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=157)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=232)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=177)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5908/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5908/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5908/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12875902 - PreCommit-HIVE-Build
> Multiline queries that have comment in middle fail when executed via "beeline
> -e"
> -
>
> Key: HIVE-17050
> URL: https://issues.apache.org/jira/browse/HIVE-17050
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-17050.1.patch
>
>
> After applying HIVE-13864, multiple line queries that have comment at the end
> of one of the middle lines fail when executed via beeline -e
> {noformat}
> $ beeline -u "" -e "select 1, --test
> > 2"
> scan complete in 3ms
> ..
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Error: Error while compiling statement: FAILED: ParseException line 1:9
> cannot recognize input near '' '' '' in selection target
> (state=42000,code=4)
> Closing: 0:
> jdbc:hive2://host-10-17-80-194.coe.cloudera.com:1/default;principal=hive/host-10-17-80-194.coe.cloudera@yshi.com;ssl=true;sslTrustStore=/certs/hive/hive-keystore.jks;trustStorePassword=cloudera
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17051) Each table metadata is requested twice during query compile
[
https://issues.apache.org/jira/browse/HIVE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076393#comment-16076393
]
Rajesh Balamohan commented on HIVE-17051:
-
Second call should be from cache. Were there multiple tables in the same query?.
> Each table metadata is requested twice during query compile
> ---
>
> Key: HIVE-17051
> URL: https://issues.apache.org/jira/browse/HIVE-17051
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Labels: performance
>
> As far as I can tell, for each table referenced in a query the metadata is
> retrieved twice during compilation:
> first call:
> {noformat}
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1320)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1275)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getTableObjectByName(SemanticAnalyzer.java:10943)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1992)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1942)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:11178)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11309)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:295)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:261)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:566)
> {noformat}
> second call:
> {noformat}
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1320)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1275)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getTableObjectByName(SemanticAnalyzer.java:10943)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1992)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1942)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1934)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:431)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11320)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:295)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:261)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:566)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17051) Each table metadata is requested twice during query compile
[
https://issues.apache.org/jira/browse/HIVE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076398#comment-16076398
]
Remus Rusanu commented on HIVE-17051:
-
This is a simple query
{noformat}
SELECT DISTINCT * FROM src;
{noformat}
If multiple tables are present (eg. JOIN) each table metadata is requested
twice.
> Each table metadata is requested twice during query compile
> ---
>
> Key: HIVE-17051
> URL: https://issues.apache.org/jira/browse/HIVE-17051
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Labels: performance
>
> As far as I can tell, for each table referenced in a query the metadata is
> retrieved twice during compilation:
> first call:
> {noformat}
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1320)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1275)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getTableObjectByName(SemanticAnalyzer.java:10943)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1992)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1942)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:11178)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11309)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:295)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:261)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:566)
> {noformat}
> second call:
> {noformat}
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1320)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1275)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getTableObjectByName(SemanticAnalyzer.java:10943)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1992)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1942)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1934)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:431)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11320)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:295)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:261)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:566)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17052) Remove logging of predicate filters
[ https://issues.apache.org/jira/browse/HIVE-17052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yibing Shi reassigned HIVE-17052: - Assignee: Yibing Shi > Remove logging of predicate filters > --- > > Key: HIVE-17052 > URL: https://issues.apache.org/jira/browse/HIVE-17052 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Yibing Shi > > HIVE-16869 added the filter predicate to the debug log of HS2, but since > these filters may contain sensitive information they should not be logged out. > The log statement should be changed back to the original form. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17052) Remove logging of predicate filters
[ https://issues.apache.org/jira/browse/HIVE-17052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yibing Shi updated HIVE-17052: -- Status: Patch Available (was: Open) > Remove logging of predicate filters > --- > > Key: HIVE-17052 > URL: https://issues.apache.org/jira/browse/HIVE-17052 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Yibing Shi > Attachments: HIVE-17052.1.patch > > > HIVE-16869 added the filter predicate to the debug log of HS2, but since > these filters may contain sensitive information they should not be logged out. > The log statement should be changed back to the original form. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17052) Remove logging of predicate filters
[ https://issues.apache.org/jira/browse/HIVE-17052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yibing Shi updated HIVE-17052: -- Attachment: HIVE-17052.1.patch Submit the patch > Remove logging of predicate filters > --- > > Key: HIVE-17052 > URL: https://issues.apache.org/jira/browse/HIVE-17052 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Yibing Shi > Attachments: HIVE-17052.1.patch > > > HIVE-16869 added the filter predicate to the debug log of HS2, but since > these filters may contain sensitive information they should not be logged out. > The log statement should be changed back to the original form. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17052) Remove logging of predicate filters
[ https://issues.apache.org/jira/browse/HIVE-17052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076404#comment-16076404 ] Barna Zsombor Klara commented on HIVE-17052: +1 LGTM > Remove logging of predicate filters > --- > > Key: HIVE-17052 > URL: https://issues.apache.org/jira/browse/HIVE-17052 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Yibing Shi > Attachments: HIVE-17052.1.patch > > > HIVE-16869 added the filter predicate to the debug log of HS2, but since > these filters may contain sensitive information they should not be logged out. > The log statement should be changed back to the original form. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17050) Multiline queries that have comment in middle fail when executed via "beeline -e"
[
https://issues.apache.org/jira/browse/HIVE-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076414#comment-16076414
]
Yibing Shi commented on HIVE-17050:
---
Error seem irrelevant. The only failed test that is possibly affected by this
change is TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1],
which doesn't contain any comment in the query script.
> Multiline queries that have comment in middle fail when executed via "beeline
> -e"
> -
>
> Key: HIVE-17050
> URL: https://issues.apache.org/jira/browse/HIVE-17050
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-17050.1.patch
>
>
> After applying HIVE-13864, multiple line queries that have comment at the end
> of one of the middle lines fail when executed via beeline -e
> {noformat}
> $ beeline -u "" -e "select 1, --test
> > 2"
> scan complete in 3ms
> ..
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Error: Error while compiling statement: FAILED: ParseException line 1:9
> cannot recognize input near '' '' '' in selection target
> (state=42000,code=4)
> Closing: 0:
> jdbc:hive2://host-10-17-80-194.coe.cloudera.com:1/default;principal=hive/host-10-17-80-194.coe.cloudera@yshi.com;ssl=true;sslTrustStore=/certs/hive/hive-keystore.jks;trustStorePassword=cloudera
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17052) Remove logging of predicate filters
[
https://issues.apache.org/jira/browse/HIVE-17052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076462#comment-16076462
]
Hive QA commented on HIVE-17052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12875915/HIVE-17052.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10818 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=232)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=102)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=177)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout
(batchId=226)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5909/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5909/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5909/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12875915 - PreCommit-HIVE-Build
> Remove logging of predicate filters
> ---
>
> Key: HIVE-17052
> URL: https://issues.apache.org/jira/browse/HIVE-17052
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Barna Zsombor Klara
>Assignee: Yibing Shi
> Attachments: HIVE-17052.1.patch
>
>
> HIVE-16869 added the filter predicate to the debug log of HS2, but since
> these filters may contain sensitive information they should not be logged out.
> The log statement should be changed back to the original form.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17018) Small table is converted to map join even the total size of small tables exceeds the threshold(hive.auto.convert.join.noconditionaltask.size)
[
https://issues.apache.org/jira/browse/HIVE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang_intel updated HIVE-17018:
Attachment: HIVE-17018.q
HIVE-17018_data_init.q
t3.txt
t3.txt, HIVE-17018_data_init.q, HIVE-17018.q is used to reproduce the case in
the comment
> Small table is converted to map join even the total size of small tables
> exceeds the threshold(hive.auto.convert.join.noconditionaltask.size)
> -
>
> Key: HIVE-17018
> URL: https://issues.apache.org/jira/browse/HIVE-17018
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-17018_data_init.q, HIVE-17018.q, t3.txt
>
>
> we use "hive.auto.convert.join.noconditionaltask.size" as the threshold. it
> means the sum of size for n-1 of the tables/partitions for a n-way join is
> smaller than it, it will be converted to a map join. for example, A join B
> join C join D join E. Big table is A(100M), small tables are
> B(10M),C(10M),D(10M),E(10M). If we set
> hive.auto.convert.join.noconditionaltask.size=20M. In current code, E,D,B
> will be converted to map join but C will not be converted to map join. In my
> understanding, because hive.auto.convert.join.noconditionaltask.size can only
> contain E and D, so C and B should not be converted to map join.
> Let's explain more why E can be converted to map join.
> in current code,
> [SparkMapJoinOptimizer#getConnectedMapJoinSize|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java#L364]
> calculates all the mapjoins in the parent path and child path. The search
> stops when encountering [UnionOperator or
> ReduceOperator|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java#L381].
> Because C is not converted to map join because {{connectedMapJoinSize +
> totalSize) > maxSize}} [see
> code|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java#L330].The
> RS before the join of C remains. When calculating whether B will be
> converted to map join, {{getConnectedMapJoinSize}} returns 0 as encountering
> [RS
> |https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java#409]
> and causes {{connectedMapJoinSize + totalSize) < maxSize}} matches.
> [~xuefuz] or [~jxiang]: can you help see whether this is a bug or not as you
> are more familiar with SparkJoinOptimizer.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16960) Hive throws an ugly error exception when HDFS sticky bit is set
[ https://issues.apache.org/jira/browse/HIVE-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-16960: --- Attachment: HIVE16960.2.patch > Hive throws an ugly error exception when HDFS sticky bit is set > --- > > Key: HIVE-16960 > URL: https://issues.apache.org/jira/browse/HIVE-16960 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Critical > Labels: newbie > Fix For: 3.0.0 > > Attachments: HIVE16960.1.patch, HIVE16960.2.patch, HIVE16960.2.patch > > > When calling LOAD DATA INPATH ... OVERWRITE INTO TABLE ... from a Hive user > other than the HDFS file owner, and the HDFS sticky bit is set, then Hive > will throw an error exception message that the file cannot be moved due to > permission issues. > Caused by: org.apache.hadoop.security.AccessControlException: Permission > denied by sticky bit setting: user=hive, > inode=sasdata-2016-04-20-17-13-43-630-e-1.dlv.bk > The permission denied is expected, but the error message does not make sense > to users + the stack trace displayed is huge. We should display a better > error message to users, and maybe provide with help information about how to > fix it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17014) Password File Encryption for HiveServer2 Client
[ https://issues.apache.org/jira/browse/HIVE-17014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Gudikov updated HIVE-17014: Attachment: PasswordFileEncryption.docx.pdf This document describes possible ways of implementing password file encryption feature > Password File Encryption for HiveServer2 Client > --- > > Key: HIVE-17014 > URL: https://issues.apache.org/jira/browse/HIVE-17014 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vlad Gudikov >Assignee: Vlad Gudikov > Fix For: 2.1.2 > > Attachments: PasswordFileEncryption.docx.pdf > > > The main point of this file is to encrypt password file that is used for > beeline connection using -w key. Any ideas or proposals would be great. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-6990) Direct SQL fails when the explicit schema setting is different from the default one
[ https://issues.apache.org/jira/browse/HIVE-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076562#comment-16076562 ] Ashutosh Chauhan commented on HIVE-6990: +1 > Direct SQL fails when the explicit schema setting is different from the > default one > --- > > Key: HIVE-6990 > URL: https://issues.apache.org/jira/browse/HIVE-6990 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.12.0, 0.14.0, 1.2.1 > Environment: hive + derby >Reporter: Bing Li >Assignee: Sergey Shelukhin > Attachments: HIVE-6990.06.patch, HIVE-6990.1.patch, > HIVE-6990.2.patch, HIVE-6990.3.patch, HIVE-6990.4.patch, HIVE-6990.5.patch > > > I got the following ERROR in hive.log > 2014-04-23 17:30:23,331 ERROR metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(1756)) - Direct SQL failed, falling > back to ORM > javax.jdo.JDODataStoreException: Error executing SQL query "select > PARTITIONS.PART_ID from PARTITIONS inner join TBLS on PARTITIONS.TBL_ID = > TBLS.TBL_ID inner join DBS on TBLS.DB_ID = DBS.DB_ID inner join > PARTITION_KEY_VALS as FILTER0 on FILTER0.PART_ID = PARTITIONS.PART_ID and > FILTER0.INTEGER_IDX = 0 where TBLS.TBL_NAME = ? and DBS.NAME = ? and > ((FILTER0.PART_KEY_VAL = ?))". > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at > org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:181) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:98) > at > org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilterInternal(ObjectStore.java:1833) > at > org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilter(ObjectStore.java:1806) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) > at java.lang.reflect.Method.invoke(Method.java:619) > at > org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124) > at com.sun.proxy.$Proxy11.getPartitionsByFilter(Unknown Source) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_by_filter(HiveMetaStore.java:3310) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) > at java.lang.reflect.Method.invoke(Method.java:619) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103) > at com.sun.proxy.$Proxy12.get_partitions_by_filter(Unknown Source) > Reproduce steps: > 1. set the following properties in hive-site.xml > > javax.jdo.mapping.Schema > HIVE > > > javax.jdo.option.ConnectionUserName > user1 > > 2. execute hive queries > hive> create table mytbl ( key int, value string); > hive> load data local inpath 'examples/files/kv1.txt' overwrite into table > mytbl; > hive> select * from mytbl; > hive> create view myview partitioned on (value) as select key, value from > mytbl where key=98; > hive> alter view myview add partition (value='val_98') partition > (value='val_xyz'); > hive> alter view myview drop partition (value='val_xyz'); -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17014) Password File Encryption for HiveServer2 Client
[ https://issues.apache.org/jira/browse/HIVE-17014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076563#comment-16076563 ] Vlad Gudikov commented on HIVE-17014: - Attached document with possible ways to implement this feature. As [~lmccay] commented in [HIVE-17014]. - "we may want to consider the use of the CredentialProvider API that will be committed soon. See [HADOOP-10607]. This isn't mutually exclusive with the password file approach as there are plans to fallback to existing password files in certain components. However, the abstraction of the API is best realized through the new Configuration.getPassword(String name) method. This will allow you to ask for a configuration item that you know is a password and it will check for an aliased credential based on the name through the CredentialProvider API. If the name is not resolved into a credential from a provider then it falls back to the config file." Would be happy to discuss this approach with other members. > Password File Encryption for HiveServer2 Client > --- > > Key: HIVE-17014 > URL: https://issues.apache.org/jira/browse/HIVE-17014 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vlad Gudikov >Assignee: Vlad Gudikov > Fix For: 2.1.2 > > Attachments: PasswordFileEncryption.docx.pdf > > > The main point of this file is to encrypt password file that is used for > beeline connection using -w key. Any ideas or proposals would be great. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16922) Typo in serde.thrift: COLLECTION_DELIM = "colelction.delim"
[ https://issues.apache.org/jira/browse/HIVE-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076574#comment-16076574 ] Bing Li commented on HIVE-16922: There is one failure caused by the fix, wil take a look at it tomorrow. https://builds.apache.org/job/PreCommit-HIVE-Build/5905/testReport/org.apache.hadoop.hive.cli/TestMiniLlapLocalCliDriver/testCliDriver_orc_create_/ > Typo in serde.thrift: COLLECTION_DELIM = "colelction.delim" > --- > > Key: HIVE-16922 > URL: https://issues.apache.org/jira/browse/HIVE-16922 > Project: Hive > Issue Type: Bug > Components: Thrift API >Reporter: Dudu Markovitz >Assignee: Bing Li > Attachments: HIVE-16922.1.patch > > > https://github.com/apache/hive/blob/master/serde/if/serde.thrift > Typo in serde.thrift: > COLLECTION_DELIM = "colelction.delim" > (*colelction* instead of *collection*) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17008) HiveMetastore.drop_database can return NPE if database does not exist
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076630#comment-16076630
]
Mohit Sabharwal commented on HIVE-17008:
{{ListenerEvent}} has the success flag, so listener implementation can choose
to ignore failed events. But {{DbNotificationListener}} and
{{HMSMetricsListener}} are ignoring the status flag. The old
{{NotificationListener}} though is looking at the flag and only logging
successful events.
This does look like something we should make consistent, not just address the
NPE. [~sushanth], [~thejas], [~akolb], do you have any thoughts ?
> HiveMetastore.drop_database can return NPE if database does not exist
> -
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Dan Burkert
> Attachments: HIVE-17008.0.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17053) Enable improved Calcite MV-based rewriting rules
[ https://issues.apache.org/jira/browse/HIVE-17053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-17053: -- > Enable improved Calcite MV-based rewriting rules > > > Key: HIVE-17053 > URL: https://issues.apache.org/jira/browse/HIVE-17053 > Project: Hive > Issue Type: New Feature > Components: Logical Optimizer, Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Integrate the rules introduced in CALCITE-1731. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17054) Expose SQL database constraints to Calcite
[ https://issues.apache.org/jira/browse/HIVE-17054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-17054: -- > Expose SQL database constraints to Calcite > -- > > Key: HIVE-17054 > URL: https://issues.apache.org/jira/browse/HIVE-17054 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Hive already has support to declare multiple SQL constraints (PRIMARY KEY, > FOREIGN KEY, UNIQUE, and NOT NULL). Although these constraints cannot be > currently enforced on the data, they can be made available to the optimizer > by using the 'RELY' keyword. > Currently, even when they are declared with the RELY keyword, they are not > exposed to Calcite. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16960) Hive throws an ugly error exception when HDFS sticky bit is set
[
https://issues.apache.org/jira/browse/HIVE-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076651#comment-16076651
]
Hive QA commented on HIVE-16960:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12875933/HIVE16960.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=238)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=238)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=233)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=177)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5910/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5910/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5910/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12875933 - PreCommit-HIVE-Build
> Hive throws an ugly error exception when HDFS sticky bit is set
> ---
>
> Key: HIVE-16960
> URL: https://issues.apache.org/jira/browse/HIVE-16960
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Critical
> Labels: newbie
> Fix For: 3.0.0
>
> Attachments: HIVE16960.1.patch, HIVE16960.2.patch, HIVE16960.2.patch
>
>
> When calling LOAD DATA INPATH ... OVERWRITE INTO TABLE ... from a Hive user
> other than the HDFS file owner, and the HDFS sticky bit is set, then Hive
> will throw an error exception message that the file cannot be moved due to
> permission issues.
> Caused by: org.apache.hadoop.security.AccessControlException: Permission
> denied by sticky bit setting: user=hive,
> inode=sasdata-2016-04-20-17-13-43-630-e-1.dlv.bk
> The permission denied is expected, but the error message does not make sense
> to users + the stack trace displayed is huge. We should display a better
> error message to users, and maybe provide with help information about how to
> fix it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17008) HiveMetastore.drop_database can return NPE if database does not exist
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076653#comment-16076653
]
Alexander Kolbasov commented on HIVE-17008:
---
There should be no events when the operation failed. This is a bug in the
notification processing.
> HiveMetastore.drop_database can return NPE if database does not exist
> -
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Dan Burkert
> Attachments: HIVE-17008.0.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17008) HiveMetastore.drop_database can return NPE if database does not exist
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076664#comment-16076664
]
Alexander Kolbasov commented on HIVE-17008:
---
Looking at the code, the actual operation is for some reason considered
successfull which means that there should be an event and it shouldn't throw
NPE.
Here is the relevant check:
https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java?utf8=✓#L1119
> HiveMetastore.drop_database can return NPE if database does not exist
> -
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Dan Burkert
> Attachments: HIVE-17008.0.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-10567) partial scan for rcfile table doesn't work for dynamic partition
[ https://issues.apache.org/jira/browse/HIVE-10567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076672#comment-16076672 ] Ashutosh Chauhan commented on HIVE-10567: - [~prasanth_j] Unfortunately partial partition spec partition(ds,hr) still needs to be specified if stats collection on all partitions is desired. We shall improve that syntax in a followup jira. In the meanwhile the bug specified here for partialscan is relevant and need to be fixed. > partial scan for rcfile table doesn't work for dynamic partition > > > Key: HIVE-10567 > URL: https://issues.apache.org/jira/browse/HIVE-10567 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.14.0, 1.0.0 >Reporter: Thomas Friedrich >Assignee: Thomas Friedrich >Priority: Minor > Labels: rcfile > Attachments: HIVE-10567.1.patch > > > HIVE-3958 added support for partial scan for RCFile. This works fine for > static partitions (for example: analyze table analyze_srcpart_partial_scan > PARTITION(ds='2008-04-08',hr=11) compute statistics partialscan). > For dynamic partition, the analyze files with an IOException > "java.io.IOException: No input paths specified in job": > hive> ANALYZE TABLE testtable PARTITION(col_varchar) COMPUTE STATISTICS > PARTIALSCAN; > java.io.IOException: No input paths specified in job > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputPaths(HiveInputFormat.java:318) > at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:459) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:624) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:616) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17034) The spark tar for itests is downloaded every time if md5sum is not installed
[ https://issues.apache.org/jira/browse/HIVE-17034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076784#comment-16076784 ] Ashutosh Chauhan commented on HIVE-17034: - +1 > The spark tar for itests is downloaded every time if md5sum is not installed > > > Key: HIVE-17034 > URL: https://issues.apache.org/jira/browse/HIVE-17034 > Project: Hive > Issue Type: Test >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-17034.1.patch > > > I think we should either skip verifying md5, or fail the build to let > developer know md5sum is required. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16989) Fix some issues identified by lgtm.com
[ https://issues.apache.org/jira/browse/HIVE-16989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Malcolm Taylor updated HIVE-16989: -- Status: In Progress (was: Patch Available) > Fix some issues identified by lgtm.com > -- > > Key: HIVE-16989 > URL: https://issues.apache.org/jira/browse/HIVE-16989 > Project: Hive > Issue Type: Improvement >Reporter: Malcolm Taylor >Assignee: Malcolm Taylor > Attachments: HIVE-16989.patch > > > [lgtm.com|https://lgtm.com] has identified a number of issues where there may > be scope for improvement. The plan is to address some of the alerts found at > [https://lgtm.com/projects/g/apache/hive/]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16989) Fix some issues identified by lgtm.com
[ https://issues.apache.org/jira/browse/HIVE-16989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Malcolm Taylor updated HIVE-16989: -- Attachment: HIVE-16989.2.patch rebased, added tests > Fix some issues identified by lgtm.com > -- > > Key: HIVE-16989 > URL: https://issues.apache.org/jira/browse/HIVE-16989 > Project: Hive > Issue Type: Improvement >Reporter: Malcolm Taylor >Assignee: Malcolm Taylor > Attachments: HIVE-16989.2.patch, HIVE-16989.patch > > > [lgtm.com|https://lgtm.com] has identified a number of issues where there may > be scope for improvement. The plan is to address some of the alerts found at > [https://lgtm.com/projects/g/apache/hive/]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16989) Fix some issues identified by lgtm.com
[ https://issues.apache.org/jira/browse/HIVE-16989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Malcolm Taylor updated HIVE-16989: -- Labels: patch (was: ) Status: Patch Available (was: In Progress) Patch 2 submitted. Please review. > Fix some issues identified by lgtm.com > -- > > Key: HIVE-16989 > URL: https://issues.apache.org/jira/browse/HIVE-16989 > Project: Hive > Issue Type: Improvement >Reporter: Malcolm Taylor >Assignee: Malcolm Taylor > Labels: patch > Attachments: HIVE-16989.2.patch, HIVE-16989.patch > > > [lgtm.com|https://lgtm.com] has identified a number of issues where there may > be scope for improvement. The plan is to address some of the alerts found at > [https://lgtm.com/projects/g/apache/hive/]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16935) Hive should strip comments from input before choosing which CommandProcessor to run.
[
https://issues.apache.org/jira/browse/HIVE-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076914#comment-16076914
]
Andrew Sherman commented on HIVE-16935:
---
Test failures look unrelated
> Hive should strip comments from input before choosing which CommandProcessor
> to run.
>
>
> Key: HIVE-16935
> URL: https://issues.apache.org/jira/browse/HIVE-16935
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-16935.1.patch, HIVE-16935.2.patch,
> HIVE-16935.3.patch, HIVE-16935.4.patch
>
>
> While using Beeswax, Hue fails to execute statement with following error:
> Error while compiling statement: FAILED: ParseException line 3:4 missing
> KW_ROLE at 'a' near 'a' line 3:5 missing EOF at '=' near 'a'
> {quote}
> -- comment
> SET a=1;
> SELECT 1;
> {quote}
> The same code works in Beeline and in Impala.
> The same code fails in CliDriver
>
> h2. Background
> Hive deals with sql comments (“-- to end of line”) in different places.
> Some clients attempt to strip comments. For example BeeLine was recently
> enhanced in https://issues.apache.org/jira/browse/HIVE-13864 to strip
> comments from multi-line commands before they are executed.
> Other clients such as Hue or Jdbc do not strip comments before sending text.
> Some tests such as TestCliDriver strip comments before running tests.
> When Hive gets a command the CommandProcessorFactory looks at the text to
> determine which CommandProcessor should handle the command. In the bug case
> the correct CommandProcessor is SetProcessor, but the comments confuse the
> CommandProcessorFactory and so the command is treated as sql. Hive’s sql
> parser understands and ignores comments, but it does not understand the set
> commands usually handled by SetProcessor and so we get the ParseException
> shown above.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16989) Fix some issues identified by lgtm.com
[
https://issues.apache.org/jira/browse/HIVE-16989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076949#comment-16076949
]
Hive QA commented on HIVE-16989:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12875948/HIVE-16989.2.patch
{color:green}SUCCESS:{color} 1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10834 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=237)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_4] (batchId=6)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=232)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=177)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5911/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5911/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5911/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12875948 - PreCommit-HIVE-Build
> Fix some issues identified by lgtm.com
> --
>
> Key: HIVE-16989
> URL: https://issues.apache.org/jira/browse/HIVE-16989
> Project: Hive
> Issue Type: Improvement
>Reporter: Malcolm Taylor
>Assignee: Malcolm Taylor
> Labels: patch
> Attachments: HIVE-16989.2.patch, HIVE-16989.patch
>
>
> [lgtm.com|https://lgtm.com] has identified a number of issues where there may
> be scope for improvement. The plan is to address some of the alerts found at
> [https://lgtm.com/projects/g/apache/hive/].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17008) HiveMetastore.drop_database can return NPE if database does not exist
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077005#comment-16077005
]
Dan Burkert commented on HIVE-17008:
My comment yesterday evening was a bit brief because I was low on time, but
here's the exact sequence of events that leads to the NPE which originally
prompted this issue. As we've discussed it's just one of many inter-related
issues in the class, but I wanted to make it clear what's happening in this
specific case:
1. Application calls {{ThriftHiveMetastore.drop_database}} with a non-existent
database name via the HMS thrift API¹.
2. In {{HiveMetaStore.drop_database_core}}, the {{db}} local variable is
[initialized to
{{null}}|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1023].
3. In {{HiveMetaStore.drop_database_core}}, the [lookup of the non-existent
database
fails|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1029],
leaving {{db}} set to {{null}} and unwinding to the {{finally}} block.
4. In {{HiveMetaSotre.drop_database_core}} {{finally}} block, a new
[{{DropDatabaseEvent}} is
created|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1139-L1143]
with the {{null}} database, and listeners are notified with this event.
5. Somewhere (I haven't traced this bit), the notification log event listener
is dereferencing the null database.
¹ Although I haven't reproduced it, it should be possible to reproduce this bug
through the Java {{HiveMetastoreClient}} API as well, but it would require
concurrent DDL operations. The {{HiveMetastoreClient}} [checks that the
database
exists|https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L851-L858]
before attempting to drop it; this is a benign TOCTOU which makes it difficult
to reproduce using that API (again, it should still possible with the right
interleavings of concurrent DDL ops). A light skimming through that class
reveals that the client is _very_ aggressive about validating state exists
before issuing DDL operations. Just my opinion, but that's a big code smell;
the client should rely on the server to validate arguments.
> HiveMetastore.drop_database can return NPE if database does not exist
> -
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Dan Burkert
> Attachments: HIVE-17008.0.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17008) HiveMetastore.drop_database can return NPE if database does not exist
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077006#comment-16077006
]
Dan Burkert commented on HIVE-17008:
My comment yesterday evening was a bit brief because I was low on time, but
here's the exact sequence of events that leads to the NPE which originally
prompted this issue. As we've discussed it's just one of many inter-related
issues in the class, but I wanted to make it clear what's happening in this
specific case:
1. Application calls {{ThriftHiveMetastore.drop_database}} with a non-existent
database name via the HMS thrift API¹.
2. In {{HiveMetaStore.drop_database_core}}, the {{db}} local variable is
[initialized to
{{null}}|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1023].
3. In {{HiveMetaStore.drop_database_core}}, the [lookup of the non-existent
database
fails|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1029],
leaving {{db}} set to {{null}} and unwinding to the {{finally}} block.
4. In {{HiveMetaSotre.drop_database_core}} {{finally}} block, a new
[{{DropDatabaseEvent}} is
created|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1139-L1143]
with the {{null}} database, and listeners are notified with this event.
5. Somewhere (I haven't traced this bit), the notification log event listener
is dereferencing the null database.
¹ Although I haven't reproduced it, it should be possible to reproduce this bug
through the Java {{HiveMetastoreClient}} API as well, but it would require
concurrent DDL operations. The {{HiveMetastoreClient}} [checks that the
database
exists|https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L851-L858]
before attempting to drop it; this is a benign TOCTOU which makes it difficult
to reproduce using that API (again, it should still possible with the right
interleavings of concurrent DDL ops). A light skimming through that class
reveals that the client is _very_ aggressive about validating state exists
before issuing DDL operations. Just my opinion, but that's a big code smell;
the client should rely on the server to validate arguments.
> HiveMetastore.drop_database can return NPE if database does not exist
> -
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Dan Burkert
> Attachments: HIVE-17008.0.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Issue Comment Deleted] (HIVE-17008) HiveMetastore.drop_database can return NPE if database does not exist
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Burkert updated HIVE-17008:
---
Comment: was deleted
(was: My comment yesterday evening was a bit brief because I was low on time,
but here's the exact sequence of events that leads to the NPE which originally
prompted this issue. As we've discussed it's just one of many inter-related
issues in the class, but I wanted to make it clear what's happening in this
specific case:
1. Application calls {{ThriftHiveMetastore.drop_database}} with a non-existent
database name via the HMS thrift API¹.
2. In {{HiveMetaStore.drop_database_core}}, the {{db}} local variable is
[initialized to
{{null}}|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1023].
3. In {{HiveMetaStore.drop_database_core}}, the [lookup of the non-existent
database
fails|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1029],
leaving {{db}} set to {{null}} and unwinding to the {{finally}} block.
4. In {{HiveMetaSotre.drop_database_core}} {{finally}} block, a new
[{{DropDatabaseEvent}} is
created|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1139-L1143]
with the {{null}} database, and listeners are notified with this event.
5. Somewhere (I haven't traced this bit), the notification log event listener
is dereferencing the null database.
¹ Although I haven't reproduced it, it should be possible to reproduce this bug
through the Java {{HiveMetastoreClient}} API as well, but it would require
concurrent DDL operations. The {{HiveMetastoreClient}} [checks that the
database
exists|https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L851-L858]
before attempting to drop it; this is a benign TOCTOU which makes it difficult
to reproduce using that API (again, it should still possible with the right
interleavings of concurrent DDL ops). A light skimming through that class
reveals that the client is _very_ aggressive about validating state exists
before issuing DDL operations. Just my opinion, but that's a big code smell;
the client should rely on the server to validate arguments.)
> HiveMetastore.drop_database can return NPE if database does not exist
> -
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Dan Burkert
> Attachments: HIVE-17008.0.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17008) HiveMetastore.drop_database can return NPE if database does not exist
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Burkert updated HIVE-17008:
---
Status: Open (was: Patch Available)
Canceling patch, since it's clear that it _is_ intended behavior to notify
listeners on failed DDL ops.
> HiveMetastore.drop_database can return NPE if database does not exist
> -
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Dan Burkert
> Attachments: HIVE-17008.0.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17008) HiveMetastore.drop_database can return NPE if database does not exist
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077005#comment-16077005
]
Dan Burkert edited comment on HIVE-17008 at 7/6/17 6:46 PM:
My comment yesterday evening was a bit brief because I was low on time, but
here's the exact sequence of events that leads to the NPE which originally
prompted this issue. As we've discussed it's just one of many inter-related
issues in the class, but I wanted to make it clear what's happening in this
specific case:
1. Application calls {{ThriftHiveMetastore.drop_database}} with a non-existent
database name via the HMS thrift API¹.
2. In {{HiveMetaStore.drop_database_core}}, the {{db}} local variable is
[initialized to
{{null}}|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1023].
3. In {{HiveMetaStore.drop_database_core}}, the [lookup of the non-existent
database
fails|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1029],
leaving {{db}} set to {{null}} and unwinding to the {{finally}} block.
4. In {{HiveMetaSotre.drop_database_core}} {{finally}} block, a new
[{{DropDatabaseEvent}} is
created|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1139-L1143]
with the {{null}} database, and listeners are notified with this event.
5. Somewhere (I haven't traced this bit), the notification log event listener
is dereferencing the null database.
¹ Although I haven't reproduced it, it should be possible to reproduce this bug
through the Java {{HiveMetastoreClient}} API as well, but it would require
concurrent DDL operations. The {{HiveMetastoreClient}} [checks that the
database
exists|https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L851-L858]
before attempting to drop it; this is a benign TOCTOU which makes it difficult
to reproduce using that API (again, it should still possible with the right
interleavings of concurrent DDL ops). A light skimming through that class
reveals that the client is very aggressive about validating state exists before
issuing DDL operations. Just my opinion, but that's a big code smell; the
client should rely on the server to validate arguments.
was (Author: danburkert):
My comment yesterday evening was a bit brief because I was low on time, but
here's the exact sequence of events that leads to the NPE which originally
prompted this issue. As we've discussed it's just one of many inter-related
issues in the class, but I wanted to make it clear what's happening in this
specific case:
1. Application calls {{ThriftHiveMetastore.drop_database}} with a non-existent
database name via the HMS thrift API¹.
2. In {{HiveMetaStore.drop_database_core}}, the {{db}} local variable is
[initialized to
{{null}}|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1023].
3. In {{HiveMetaStore.drop_database_core}}, the [lookup of the non-existent
database
fails|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1029],
leaving {{db}} set to {{null}} and unwinding to the {{finally}} block.
4. In {{HiveMetaSotre.drop_database_core}} {{finally}} block, a new
[{{DropDatabaseEvent}} is
created|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1139-L1143]
with the {{null}} database, and listeners are notified with this event.
5. Somewhere (I haven't traced this bit), the notification log event listener
is dereferencing the null database.
¹ Although I haven't reproduced it, it should be possible to reproduce this bug
through the Java {{HiveMetastoreClient}} API as well, but it would require
concurrent DDL operations. The {{HiveMetastoreClient}} [checks that the
database
exists|https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L851-L858]
before attempting to drop it; this is a benign TOCTOU which makes it difficult
to reproduce using that API (again, it should still possible with the right
interleavings of concurrent DDL ops). A light skimming through that class
reveals that the client is _very_ aggressive about validating state exists
before issuing DDL operations. Just my opinion, but that's a big code smell;
the client should rely on the server to validate arguments.
> HiveMetastore.drop_database can return NPE if database does not exist
[jira] [Comment Edited] (HIVE-17008) HiveMetastore.drop_database can return NPE if database does not exist
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077008#comment-16077008
]
Dan Burkert edited comment on HIVE-17008 at 7/6/17 6:46 PM:
Canceling patch, since it's clear that it is intended behavior to notify
listeners on failed DDL ops.
was (Author: danburkert):
Canceling patch, since it's clear that it _is_ intended behavior to notify
listeners on failed DDL ops.
> HiveMetastore.drop_database can return NPE if database does not exist
> -
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Dan Burkert
> Attachments: HIVE-17008.0.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17056) Flaky Test: TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
[ https://issues.apache.org/jira/browse/HIVE-17056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-17056: --- Summary: Flaky Test: TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic] (was: TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]) > Flaky Test: TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic] > -- > > Key: HIVE-17056 > URL: https://issues.apache.org/jira/browse/HIVE-17056 > Project: Hive > Issue Type: Sub-task >Reporter: Janaki Lahorani > > Client Execution succeeded but contained differences (error code = 1) after > executing orc_ppd_basic.q > 1287a1288 > > Stage-1 LLAP IO COUNTERS: > 1317a1319 > > Stage-1 LLAP IO COUNTERS: > 1338a1341 > > Stage-1 LLAP IO COUNTERS: > 1342a1346 > > Stage-1 LLAP IO COUNTERS: -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17057) Flaky test: TestHCatClient.testTableSchemaPropagation,testPartitionRegistrationWithCustomSchema,testPartitionSpecRegistrationWithCustomSchema
[ https://issues.apache.org/jira/browse/HIVE-17057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077086#comment-16077086 ] Janaki Lahorani commented on HIVE-17057: Unexpected exception! org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred while processing HCat request : MetaException while updating table schema.. Cause : MetaException(message:Persistence Manager has been closed) > Flaky test: > TestHCatClient.testTableSchemaPropagation,testPartitionRegistrationWithCustomSchema,testPartitionSpecRegistrationWithCustomSchema > - > > Key: HIVE-17057 > URL: https://issues.apache.org/jira/browse/HIVE-17057 > Project: Hive > Issue Type: Sub-task >Reporter: Janaki Lahorani > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16960) Hive throws an ugly error exception when HDFS sticky bit is set
[ https://issues.apache.org/jira/browse/HIVE-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077087#comment-16077087 ] Janaki Lahorani commented on HIVE-16960: These test errors are not caused by this patch. Please refer to https://issues.apache.org/jira/browse/HIVE-15058. > Hive throws an ugly error exception when HDFS sticky bit is set > --- > > Key: HIVE-16960 > URL: https://issues.apache.org/jira/browse/HIVE-16960 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Critical > Labels: newbie > Fix For: 3.0.0 > > Attachments: HIVE16960.1.patch, HIVE16960.2.patch, HIVE16960.2.patch > > > When calling LOAD DATA INPATH ... OVERWRITE INTO TABLE ... from a Hive user > other than the HDFS file owner, and the HDFS sticky bit is set, then Hive > will throw an error exception message that the file cannot be moved due to > permission issues. > Caused by: org.apache.hadoop.security.AccessControlException: Permission > denied by sticky bit setting: user=hive, > inode=sasdata-2016-04-20-17-13-43-630-e-1.dlv.bk > The permission denied is expected, but the error message does not make sense > to users + the stack trace displayed is huge. We should display a better > error message to users, and maybe provide with help information about how to > fix it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16935) Hive should strip comments from input before choosing which CommandProcessor to run.
[
https://issues.apache.org/jira/browse/HIVE-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077089#comment-16077089
]
Sahil Takiar commented on HIVE-16935:
-
[~vihangk1] could you take a quick look at the RB?
> Hive should strip comments from input before choosing which CommandProcessor
> to run.
>
>
> Key: HIVE-16935
> URL: https://issues.apache.org/jira/browse/HIVE-16935
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-16935.1.patch, HIVE-16935.2.patch,
> HIVE-16935.3.patch, HIVE-16935.4.patch
>
>
> While using Beeswax, Hue fails to execute statement with following error:
> Error while compiling statement: FAILED: ParseException line 3:4 missing
> KW_ROLE at 'a' near 'a' line 3:5 missing EOF at '=' near 'a'
> {quote}
> -- comment
> SET a=1;
> SELECT 1;
> {quote}
> The same code works in Beeline and in Impala.
> The same code fails in CliDriver
>
> h2. Background
> Hive deals with sql comments (“-- to end of line”) in different places.
> Some clients attempt to strip comments. For example BeeLine was recently
> enhanced in https://issues.apache.org/jira/browse/HIVE-13864 to strip
> comments from multi-line commands before they are executed.
> Other clients such as Hue or Jdbc do not strip comments before sending text.
> Some tests such as TestCliDriver strip comments before running tests.
> When Hive gets a command the CommandProcessorFactory looks at the text to
> determine which CommandProcessor should handle the command. In the bug case
> the correct CommandProcessor is SetProcessor, but the comments confuse the
> CommandProcessorFactory and so the command is treated as sql. Hive’s sql
> parser understands and ignores comments, but it does not understand the set
> commands usually handled by SetProcessor and so we get the ParseException
> shown above.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16935) Hive should strip comments from input before choosing which CommandProcessor to run.
[
https://issues.apache.org/jira/browse/HIVE-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077157#comment-16077157
]
Sergio Peña commented on HIVE-16935:
The code looks very good [~asherman]
1
> Hive should strip comments from input before choosing which CommandProcessor
> to run.
>
>
> Key: HIVE-16935
> URL: https://issues.apache.org/jira/browse/HIVE-16935
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-16935.1.patch, HIVE-16935.2.patch,
> HIVE-16935.3.patch, HIVE-16935.4.patch
>
>
> While using Beeswax, Hue fails to execute statement with following error:
> Error while compiling statement: FAILED: ParseException line 3:4 missing
> KW_ROLE at 'a' near 'a' line 3:5 missing EOF at '=' near 'a'
> {quote}
> -- comment
> SET a=1;
> SELECT 1;
> {quote}
> The same code works in Beeline and in Impala.
> The same code fails in CliDriver
>
> h2. Background
> Hive deals with sql comments (“-- to end of line”) in different places.
> Some clients attempt to strip comments. For example BeeLine was recently
> enhanced in https://issues.apache.org/jira/browse/HIVE-13864 to strip
> comments from multi-line commands before they are executed.
> Other clients such as Hue or Jdbc do not strip comments before sending text.
> Some tests such as TestCliDriver strip comments before running tests.
> When Hive gets a command the CommandProcessorFactory looks at the text to
> determine which CommandProcessor should handle the command. In the bug case
> the correct CommandProcessor is SetProcessor, but the comments confuse the
> CommandProcessorFactory and so the command is treated as sql. Hive’s sql
> parser understands and ignores comments, but it does not understand the set
> commands usually handled by SetProcessor and so we get the ParseException
> shown above.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-16935) Hive should strip comments from input before choosing which CommandProcessor to run.
[
https://issues.apache.org/jira/browse/HIVE-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077157#comment-16077157
]
Sergio Peña edited comment on HIVE-16935 at 7/6/17 8:23 PM:
The code looks very good [~asherman]
+1
was (Author: spena):
The code looks very good [~asherman]
1
> Hive should strip comments from input before choosing which CommandProcessor
> to run.
>
>
> Key: HIVE-16935
> URL: https://issues.apache.org/jira/browse/HIVE-16935
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-16935.1.patch, HIVE-16935.2.patch,
> HIVE-16935.3.patch, HIVE-16935.4.patch
>
>
> While using Beeswax, Hue fails to execute statement with following error:
> Error while compiling statement: FAILED: ParseException line 3:4 missing
> KW_ROLE at 'a' near 'a' line 3:5 missing EOF at '=' near 'a'
> {quote}
> -- comment
> SET a=1;
> SELECT 1;
> {quote}
> The same code works in Beeline and in Impala.
> The same code fails in CliDriver
>
> h2. Background
> Hive deals with sql comments (“-- to end of line”) in different places.
> Some clients attempt to strip comments. For example BeeLine was recently
> enhanced in https://issues.apache.org/jira/browse/HIVE-13864 to strip
> comments from multi-line commands before they are executed.
> Other clients such as Hue or Jdbc do not strip comments before sending text.
> Some tests such as TestCliDriver strip comments before running tests.
> When Hive gets a command the CommandProcessorFactory looks at the text to
> determine which CommandProcessor should handle the command. In the bug case
> the correct CommandProcessor is SetProcessor, but the comments confuse the
> CommandProcessorFactory and so the command is treated as sql. Hive’s sql
> parser understands and ignores comments, but it does not understand the set
> commands usually handled by SetProcessor and so we get the ParseException
> shown above.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16935) Hive should strip comments from input before choosing which CommandProcessor to run.
[
https://issues.apache.org/jira/browse/HIVE-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-16935:
---
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Thanks [~asherman]. I committed this to master.
> Hive should strip comments from input before choosing which CommandProcessor
> to run.
>
>
> Key: HIVE-16935
> URL: https://issues.apache.org/jira/browse/HIVE-16935
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Fix For: 3.0.0
>
> Attachments: HIVE-16935.1.patch, HIVE-16935.2.patch,
> HIVE-16935.3.patch, HIVE-16935.4.patch
>
>
> While using Beeswax, Hue fails to execute statement with following error:
> Error while compiling statement: FAILED: ParseException line 3:4 missing
> KW_ROLE at 'a' near 'a' line 3:5 missing EOF at '=' near 'a'
> {quote}
> -- comment
> SET a=1;
> SELECT 1;
> {quote}
> The same code works in Beeline and in Impala.
> The same code fails in CliDriver
>
> h2. Background
> Hive deals with sql comments (“-- to end of line”) in different places.
> Some clients attempt to strip comments. For example BeeLine was recently
> enhanced in https://issues.apache.org/jira/browse/HIVE-13864 to strip
> comments from multi-line commands before they are executed.
> Other clients such as Hue or Jdbc do not strip comments before sending text.
> Some tests such as TestCliDriver strip comments before running tests.
> When Hive gets a command the CommandProcessorFactory looks at the text to
> determine which CommandProcessor should handle the command. In the bug case
> the correct CommandProcessor is SetProcessor, but the comments confuse the
> CommandProcessorFactory and so the command is treated as sql. Hive’s sql
> parser understands and ignores comments, but it does not understand the set
> commands usually handled by SetProcessor and so we get the ParseException
> shown above.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-12767: -- Labels: TODOC2.4 (was: ) > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Labels: TODOC2.4 > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16935) Hive should strip comments from input before choosing which CommandProcessor to run.
[
https://issues.apache.org/jira/browse/HIVE-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077225#comment-16077225
]
Andrew Sherman commented on HIVE-16935:
---
Thank you [~spena]
> Hive should strip comments from input before choosing which CommandProcessor
> to run.
>
>
> Key: HIVE-16935
> URL: https://issues.apache.org/jira/browse/HIVE-16935
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Fix For: 3.0.0
>
> Attachments: HIVE-16935.1.patch, HIVE-16935.2.patch,
> HIVE-16935.3.patch, HIVE-16935.4.patch
>
>
> While using Beeswax, Hue fails to execute statement with following error:
> Error while compiling statement: FAILED: ParseException line 3:4 missing
> KW_ROLE at 'a' near 'a' line 3:5 missing EOF at '=' near 'a'
> {quote}
> -- comment
> SET a=1;
> SELECT 1;
> {quote}
> The same code works in Beeline and in Impala.
> The same code fails in CliDriver
>
> h2. Background
> Hive deals with sql comments (“-- to end of line”) in different places.
> Some clients attempt to strip comments. For example BeeLine was recently
> enhanced in https://issues.apache.org/jira/browse/HIVE-13864 to strip
> comments from multi-line commands before they are executed.
> Other clients such as Hue or Jdbc do not strip comments before sending text.
> Some tests such as TestCliDriver strip comments before running tests.
> When Hive gets a command the CommandProcessorFactory looks at the text to
> determine which CommandProcessor should handle the command. In the bug case
> the correct CommandProcessor is SetProcessor, but the comments confuse the
> CommandProcessorFactory and so the command is treated as sql. Hive’s sql
> parser understands and ignores comments, but it does not understand the set
> commands usually handled by SetProcessor and so we get the ParseException
> shown above.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17022) Add mode in lock debug statements
[ https://issues.apache.org/jira/browse/HIVE-17022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-17022: --- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks Naveen Gangam for the review. > Add mode in lock debug statements > - > > Key: HIVE-17022 > URL: https://issues.apache.org/jira/browse/HIVE-17022 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-17022.1.patch, HIVE-17022.patch > > > Currently, lock debug statements print IMPLICIT/EXPLICIT as lock mode, > whereas SHARED/EXCLUSIVE/SEMI_SHARED are more useful > when debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077227#comment-16077227 ] Lefty Leverenz commented on HIVE-12767: --- Added a TODOC2.4 label in case we need to revert the documentation (3 wikidocs listed in comment 19/May/17). But let's wait a while and see how the issue is solved. > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Labels: TODOC2.4 > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16962) Better error msg for Hive on Spark in case user cancels query and closes session
[
https://issues.apache.org/jira/browse/HIVE-16962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077260#comment-16077260
]
Lefty Leverenz commented on HIVE-16962:
---
Thanks Xuefu, and I certainly agree about all the pending releases. (I keep a
crib sheet handy.)
> Better error msg for Hive on Spark in case user cancels query and closes
> session
>
>
> Key: HIVE-16962
> URL: https://issues.apache.org/jira/browse/HIVE-16962
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Fix For: 3.0.0
>
> Attachments: HIVE-16962.2.patch, HIVE-16962.patch, HIVE-16962.patch
>
>
> In case user cancels a query and closes the session, Hive marks the query as
> failed. However, the error message is a little confusing. It still says:
> {quote}
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create spark
> client. This is likely because the queue you assigned to does not have free
> resource at the moment to start the job. Please check your queue usage and
> try the query again later.
> {quote}
> followed by some InterruptedException.
> Ideally, the error should clearly indicates the fact that user cancels the
> execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16751) Support different types for grouping columns in GroupBy Druid queries
[ https://issues.apache.org/jira/browse/HIVE-16751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077289#comment-16077289 ] Lefty Leverenz commented on HIVE-16751: --- [~jcamachorodriguez], does this need any user documentation? > Support different types for grouping columns in GroupBy Druid queries > - > > Key: HIVE-16751 > URL: https://issues.apache.org/jira/browse/HIVE-16751 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 3.0.0 > > Attachments: HIVE-16751.patch > > > Calcite 1.13 pushes EXTRACT and FLOOR function to Druid as an extraction > function (cf CALCITE-1758). Originally, we were assuming that all group by > columns in a druid query were of STRING type; however, this will not true > anymore (result of EXTRACT is an INT and result of FLOOR a TIMESTAMP). > When we upgrade to Calcite 1.13, we will need to extend the DruidSerDe to > handle these functions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17048) Pass HiveOperation info to HiveSemanticAnalyzerHook through HiveSemanticAnalyzerHookContext
[
https://issues.apache.org/jira/browse/HIVE-17048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-17048:
Attachment: HIVE-17048.2.patch
> Pass HiveOperation info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext
> ---
>
> Key: HIVE-17048
> URL: https://issues.apache.org/jira/browse/HIVE-17048
> Project: Hive
> Issue Type: Improvement
> Components: Hooks
>Affects Versions: 2.1.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17048.1.patch, HIVE-17048.2.patch
>
>
> Currently hive passes the following info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext (see
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L553).
> But the operation type (HiveOperation) is also needed in some cases, e.g.,
> when integrating with Sentry.
> {noformat}
> hookCtx.setConf(conf);
> hookCtx.setUserName(userName);
> hookCtx.setIpAddress(SessionState.get().getUserIpAddress());
> hookCtx.setCommand(command);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17048) Pass HiveOperation info to HiveSemanticAnalyzerHook through HiveSemanticAnalyzerHookContext
[
https://issues.apache.org/jira/browse/HIVE-17048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-17048:
Status: Patch Available (was: In Progress)
> Pass HiveOperation info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext
> ---
>
> Key: HIVE-17048
> URL: https://issues.apache.org/jira/browse/HIVE-17048
> Project: Hive
> Issue Type: Improvement
> Components: Hooks
>Affects Versions: 2.1.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17048.1.patch, HIVE-17048.2.patch
>
>
> Currently hive passes the following info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext (see
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L553).
> But the operation type (HiveOperation) is also needed in some cases, e.g.,
> when integrating with Sentry.
> {noformat}
> hookCtx.setConf(conf);
> hookCtx.setUserName(userName);
> hookCtx.setIpAddress(SessionState.get().getUserIpAddress());
> hookCtx.setCommand(command);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17048) Pass HiveOperation info to HiveSemanticAnalyzerHook through HiveSemanticAnalyzerHookContext
[
https://issues.apache.org/jira/browse/HIVE-17048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-17048:
Status: In Progress (was: Patch Available)
> Pass HiveOperation info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext
> ---
>
> Key: HIVE-17048
> URL: https://issues.apache.org/jira/browse/HIVE-17048
> Project: Hive
> Issue Type: Improvement
> Components: Hooks
>Affects Versions: 2.1.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17048.1.patch, HIVE-17048.2.patch
>
>
> Currently hive passes the following info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext (see
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L553).
> But the operation type (HiveOperation) is also needed in some cases, e.g.,
> when integrating with Sentry.
> {noformat}
> hookCtx.setConf(conf);
> hookCtx.setUserName(userName);
> hookCtx.setIpAddress(SessionState.get().getUserIpAddress());
> hookCtx.setCommand(command);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17048) Pass HiveOperation info to HiveSemanticAnalyzerHook through HiveSemanticAnalyzerHookContext
[
https://issues.apache.org/jira/browse/HIVE-17048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077309#comment-16077309
]
Aihua Xu commented on HIVE-17048:
-
[~mohitsabharwal] Attached the patch-2. Updated the unit tests as you
mentioned. I also replaced the deprecated Assert references. Can you take a
look?
> Pass HiveOperation info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext
> ---
>
> Key: HIVE-17048
> URL: https://issues.apache.org/jira/browse/HIVE-17048
> Project: Hive
> Issue Type: Improvement
> Components: Hooks
>Affects Versions: 2.1.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17048.1.patch, HIVE-17048.2.patch
>
>
> Currently hive passes the following info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext (see
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L553).
> But the operation type (HiveOperation) is also needed in some cases, e.g.,
> when integrating with Sentry.
> {noformat}
> hookCtx.setConf(conf);
> hookCtx.setUserName(userName);
> hookCtx.setIpAddress(SessionState.get().getUserIpAddress());
> hookCtx.setCommand(command);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17050) Multiline queries that have comment in middle fail when executed via "beeline -e"
[
https://issues.apache.org/jira/browse/HIVE-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077344#comment-16077344
]
Andrew Sherman commented on HIVE-17050:
---
Hi [~Yibing]
please note that as part of the fix for
https://issues.apache.org/jira/browse/HIVE-16935
I moved the method removeComments() from
beeline/src/java/org/apache/hive/beeline/Commands.java to
common/src/java/org/apache/hive/common/util/HiveStringUtils.java and also
changed it.
I think this means you will have to redo your change, sorry.
Let me know if I can help in any way.
> Multiline queries that have comment in middle fail when executed via "beeline
> -e"
> -
>
> Key: HIVE-17050
> URL: https://issues.apache.org/jira/browse/HIVE-17050
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-17050.1.patch
>
>
> After applying HIVE-13864, multiple line queries that have comment at the end
> of one of the middle lines fail when executed via beeline -e
> {noformat}
> $ beeline -u "" -e "select 1, --test
> > 2"
> scan complete in 3ms
> ..
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Error: Error while compiling statement: FAILED: ParseException line 1:9
> cannot recognize input near '' '' '' in selection target
> (state=42000,code=4)
> Closing: 0:
> jdbc:hive2://host-10-17-80-194.coe.cloudera.com:1/default;principal=hive/host-10-17-80-194.coe.cloudera@yshi.com;ssl=true;sslTrustStore=/certs/hive/hive-keystore.jks;trustStorePassword=cloudera
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17048) Pass HiveOperation info to HiveSemanticAnalyzerHook through HiveSemanticAnalyzerHookContext
[
https://issues.apache.org/jira/browse/HIVE-17048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077382#comment-16077382
]
Hive QA commented on HIVE-17048:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12875997/HIVE-17048.2.patch
{color:green}SUCCESS:{color} 1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10834 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=232)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=177)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5912/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5912/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5912/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12875997 - PreCommit-HIVE-Build
> Pass HiveOperation info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext
> ---
>
> Key: HIVE-17048
> URL: https://issues.apache.org/jira/browse/HIVE-17048
> Project: Hive
> Issue Type: Improvement
> Components: Hooks
>Affects Versions: 2.1.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17048.1.patch, HIVE-17048.2.patch
>
>
> Currently hive passes the following info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext (see
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L553).
> But the operation type (HiveOperation) is also needed in some cases, e.g.,
> when integrating with Sentry.
> {noformat}
> hookCtx.setConf(conf);
> hookCtx.setUserName(userName);
> hookCtx.setIpAddress(SessionState.get().getUserIpAddress());
> hookCtx.setCommand(command);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17050) Multiline queries that have comment in middle fail when executed via "beeline -e"
[
https://issues.apache.org/jira/browse/HIVE-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077393#comment-16077393
]
Yibing Shi commented on HIVE-17050:
---
[~asherman], your change has covered what my change does. So I just added a few
more tests in this JIRA.
> Multiline queries that have comment in middle fail when executed via "beeline
> -e"
> -
>
> Key: HIVE-17050
> URL: https://issues.apache.org/jira/browse/HIVE-17050
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-17050.1.patch, HIVE-17050.2.patch
>
>
> After applying HIVE-13864, multiple line queries that have comment at the end
> of one of the middle lines fail when executed via beeline -e
> {noformat}
> $ beeline -u "" -e "select 1, --test
> > 2"
> scan complete in 3ms
> ..
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Error: Error while compiling statement: FAILED: ParseException line 1:9
> cannot recognize input near '' '' '' in selection target
> (state=42000,code=4)
> Closing: 0:
> jdbc:hive2://host-10-17-80-194.coe.cloudera.com:1/default;principal=hive/host-10-17-80-194.coe.cloudera@yshi.com;ssl=true;sslTrustStore=/certs/hive/hive-keystore.jks;trustStorePassword=cloudera
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17050) Multiline queries that have comment in middle fail when executed via "beeline -e"
[
https://issues.apache.org/jira/browse/HIVE-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated HIVE-17050:
--
Attachment: HIVE-17050.2.patch
> Multiline queries that have comment in middle fail when executed via "beeline
> -e"
> -
>
> Key: HIVE-17050
> URL: https://issues.apache.org/jira/browse/HIVE-17050
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-17050.1.patch, HIVE-17050.2.patch
>
>
> After applying HIVE-13864, multiple line queries that have comment at the end
> of one of the middle lines fail when executed via beeline -e
> {noformat}
> $ beeline -u "" -e "select 1, --test
> > 2"
> scan complete in 3ms
> ..
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Error: Error while compiling statement: FAILED: ParseException line 1:9
> cannot recognize input near '' '' '' in selection target
> (state=42000,code=4)
> Closing: 0:
> jdbc:hive2://host-10-17-80-194.coe.cloudera.com:1/default;principal=hive/host-10-17-80-194.coe.cloudera@yshi.com;ssl=true;sslTrustStore=/certs/hive/hive-keystore.jks;trustStorePassword=cloudera
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17050) Multiline queries that have comment in middle fail when executed via "beeline -e"
[
https://issues.apache.org/jira/browse/HIVE-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077410#comment-16077410
]
Andrew Sherman commented on HIVE-17050:
---
[~Yibing] OK that is good, thanks for adding more tests
> Multiline queries that have comment in middle fail when executed via "beeline
> -e"
> -
>
> Key: HIVE-17050
> URL: https://issues.apache.org/jira/browse/HIVE-17050
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-17050.1.patch, HIVE-17050.2.patch
>
>
> After applying HIVE-13864, multiple line queries that have comment at the end
> of one of the middle lines fail when executed via beeline -e
> {noformat}
> $ beeline -u "" -e "select 1, --test
> > 2"
> scan complete in 3ms
> ..
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Error: Error while compiling statement: FAILED: ParseException line 1:9
> cannot recognize input near '' '' '' in selection target
> (state=42000,code=4)
> Closing: 0:
> jdbc:hive2://host-10-17-80-194.coe.cloudera.com:1/default;principal=hive/host-10-17-80-194.coe.cloudera@yshi.com;ssl=true;sslTrustStore=/certs/hive/hive-keystore.jks;trustStorePassword=cloudera
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17050) Multiline queries that have comment in middle fail when executed via "beeline -e"
[
https://issues.apache.org/jira/browse/HIVE-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077450#comment-16077450
]
Hive QA commented on HIVE-17050:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12876005/HIVE-17050.2.patch
{color:green}SUCCESS:{color} 1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10834 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=237)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=157)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=232)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=177)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5913/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5913/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5913/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12876005 - PreCommit-HIVE-Build
> Multiline queries that have comment in middle fail when executed via "beeline
> -e"
> -
>
> Key: HIVE-17050
> URL: https://issues.apache.org/jira/browse/HIVE-17050
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-17050.1.patch, HIVE-17050.2.patch
>
>
> After applying HIVE-13864, multiple line queries that have comment at the end
> of one of the middle lines fail when executed via beeline -e
> {noformat}
> $ beeline -u "" -e "select 1, --test
> > 2"
> scan complete in 3ms
> ..
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Error: Error while compiling statement: FAILED: ParseException line 1:9
> cannot recognize input near '' '' '' in selection target
> (state=42000,code=4)
> Closing: 0:
> jdbc:hive2://host-10-17-80-194.coe.cloudera.com:1/default;principal=hive/host-10-17-80-194.coe.cloudera@yshi.com;ssl=true;sslTrustStore=/certs/hive/hive-keystore.jks;trustStorePassword=cloudera
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17034) The spark tar for itests is downloaded every time if md5sum is not installed
[ https://issues.apache.org/jira/browse/HIVE-17034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17034: -- Component/s: Spark > The spark tar for itests is downloaded every time if md5sum is not installed > > > Key: HIVE-17034 > URL: https://issues.apache.org/jira/browse/HIVE-17034 > Project: Hive > Issue Type: Test > Components: Spark >Reporter: Rui Li >Assignee: Rui Li >Priority: Trivial > Attachments: HIVE-17034.1.patch > > > I think we should either skip verifying md5, or fail the build to let > developer know md5sum is required. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17034) The spark tar for itests is downloaded every time if md5sum is not installed
[ https://issues.apache.org/jira/browse/HIVE-17034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17034: -- Priority: Trivial (was: Major) > The spark tar for itests is downloaded every time if md5sum is not installed > > > Key: HIVE-17034 > URL: https://issues.apache.org/jira/browse/HIVE-17034 > Project: Hive > Issue Type: Test > Components: Spark >Reporter: Rui Li >Assignee: Rui Li >Priority: Trivial > Attachments: HIVE-17034.1.patch > > > I think we should either skip verifying md5, or fail the build to let > developer know md5sum is required. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17034) The spark tar for itests is downloaded every time if md5sum is not installed
[ https://issues.apache.org/jira/browse/HIVE-17034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17034: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks Ashutosh. > The spark tar for itests is downloaded every time if md5sum is not installed > > > Key: HIVE-17034 > URL: https://issues.apache.org/jira/browse/HIVE-17034 > Project: Hive > Issue Type: Test > Components: Spark >Reporter: Rui Li >Assignee: Rui Li >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-17034.1.patch > > > I think we should either skip verifying md5, or fail the build to let > developer know md5sum is required. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17048) Pass HiveOperation info to HiveSemanticAnalyzerHook through HiveSemanticAnalyzerHookContext
[
https://issues.apache.org/jira/browse/HIVE-17048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077493#comment-16077493
]
Mohit Sabharwal commented on HIVE-17048:
1, LGTM
> Pass HiveOperation info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext
> ---
>
> Key: HIVE-17048
> URL: https://issues.apache.org/jira/browse/HIVE-17048
> Project: Hive
> Issue Type: Improvement
> Components: Hooks
>Affects Versions: 2.1.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17048.1.patch, HIVE-17048.2.patch
>
>
> Currently hive passes the following info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext (see
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L553).
> But the operation type (HiveOperation) is also needed in some cases, e.g.,
> when integrating with Sentry.
> {noformat}
> hookCtx.setConf(conf);
> hookCtx.setUserName(userName);
> hookCtx.setIpAddress(SessionState.get().getUserIpAddress());
> hookCtx.setCommand(command);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17010) Fix the overflow problem of Long type in SetSparkReducerParallelism
[
https://issues.apache.org/jira/browse/HIVE-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077497#comment-16077497
]
Rui Li commented on HIVE-17010:
---
Hi [~kellyzly], I think it should be {{numberOfBytes =
StatsUtils.safeAdd(numberOfBytes, sibling.getStatistics().getDataSize());}}
right?
And thanks [~csun] for the detailed analysis. Seems the estimation logic
deserves some improvement.
> Fix the overflow problem of Long type in SetSparkReducerParallelism
> ---
>
> Key: HIVE-17010
> URL: https://issues.apache.org/jira/browse/HIVE-17010
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-17010.1.patch, HIVE-17010.2.patch
>
>
> We use
> [numberOfByteshttps://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L129]
> to collect the numberOfBytes of sibling of specified RS. We use Long type
> and it happens overflow when the data is too big. After happening this
> situation, the parallelism is decided by
> [sparkMemoryAndCores.getSecond()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L184]
> if spark.dynamic.allocation.enabled is true, sparkMemoryAndCores.getSecond
> is a dymamic value which is decided by spark runtime. For example, the value
> of sparkMemoryAndCores.getSecond is 5 or 15 randomly. There is possibility
> that the value may be 1. The may problem here is the overflow of addition of
> Long type. You can reproduce the overflow problem by following code
> {code}
> public static void main(String[] args) {
> long a1= 9223372036854775807L;
> long a2=1022672;
> long res = a1+a2;
> System.out.println(res); //-9223372036853753137
> BigInteger b1= BigInteger.valueOf(a1);
> BigInteger b2 = BigInteger.valueOf(a2);
> BigInteger bigRes = b1.add(b2);
> System.out.println(bigRes); //9223372036855798479
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17010) Fix the overflow problem of Long type in SetSparkReducerParallelism
[
https://issues.apache.org/jira/browse/HIVE-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077513#comment-16077513
]
liyunzhang_intel commented on HIVE-17010:
-
[~lirui]: thanks for your catch, will update the patch soon
> Fix the overflow problem of Long type in SetSparkReducerParallelism
> ---
>
> Key: HIVE-17010
> URL: https://issues.apache.org/jira/browse/HIVE-17010
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-17010.1.patch, HIVE-17010.2.patch
>
>
> We use
> [numberOfByteshttps://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L129]
> to collect the numberOfBytes of sibling of specified RS. We use Long type
> and it happens overflow when the data is too big. After happening this
> situation, the parallelism is decided by
> [sparkMemoryAndCores.getSecond()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L184]
> if spark.dynamic.allocation.enabled is true, sparkMemoryAndCores.getSecond
> is a dymamic value which is decided by spark runtime. For example, the value
> of sparkMemoryAndCores.getSecond is 5 or 15 randomly. There is possibility
> that the value may be 1. The may problem here is the overflow of addition of
> Long type. You can reproduce the overflow problem by following code
> {code}
> public static void main(String[] args) {
> long a1= 9223372036854775807L;
> long a2=1022672;
> long res = a1+a2;
> System.out.println(res); //-9223372036853753137
> BigInteger b1= BigInteger.valueOf(a1);
> BigInteger b2 = BigInteger.valueOf(a2);
> BigInteger bigRes = b1.add(b2);
> System.out.println(bigRes); //9223372036855798479
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17010) Fix the overflow problem of Long type in SetSparkReducerParallelism
[
https://issues.apache.org/jira/browse/HIVE-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang_intel updated HIVE-17010:
Attachment: HIVE-17010.3.patch
[~lirui]: help review HIVE-17010.3.patch.
> Fix the overflow problem of Long type in SetSparkReducerParallelism
> ---
>
> Key: HIVE-17010
> URL: https://issues.apache.org/jira/browse/HIVE-17010
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-17010.1.patch, HIVE-17010.2.patch,
> HIVE-17010.3.patch
>
>
> We use
> [numberOfByteshttps://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L129]
> to collect the numberOfBytes of sibling of specified RS. We use Long type
> and it happens overflow when the data is too big. After happening this
> situation, the parallelism is decided by
> [sparkMemoryAndCores.getSecond()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L184]
> if spark.dynamic.allocation.enabled is true, sparkMemoryAndCores.getSecond
> is a dymamic value which is decided by spark runtime. For example, the value
> of sparkMemoryAndCores.getSecond is 5 or 15 randomly. There is possibility
> that the value may be 1. The may problem here is the overflow of addition of
> Long type. You can reproduce the overflow problem by following code
> {code}
> public static void main(String[] args) {
> long a1= 9223372036854775807L;
> long a2=1022672;
> long res = a1+a2;
> System.out.println(res); //-9223372036853753137
> BigInteger b1= BigInteger.valueOf(a1);
> BigInteger b2 = BigInteger.valueOf(a2);
> BigInteger bigRes = b1.add(b2);
> System.out.println(bigRes); //9223372036855798479
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17010) Fix the overflow problem of Long type in SetSparkReducerParallelism
[
https://issues.apache.org/jira/browse/HIVE-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077534#comment-16077534
]
Rui Li commented on HIVE-17010:
---
1
> Fix the overflow problem of Long type in SetSparkReducerParallelism
> ---
>
> Key: HIVE-17010
> URL: https://issues.apache.org/jira/browse/HIVE-17010
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-17010.1.patch, HIVE-17010.2.patch,
> HIVE-17010.3.patch
>
>
> We use
> [numberOfByteshttps://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L129]
> to collect the numberOfBytes of sibling of specified RS. We use Long type
> and it happens overflow when the data is too big. After happening this
> situation, the parallelism is decided by
> [sparkMemoryAndCores.getSecond()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L184]
> if spark.dynamic.allocation.enabled is true, sparkMemoryAndCores.getSecond
> is a dymamic value which is decided by spark runtime. For example, the value
> of sparkMemoryAndCores.getSecond is 5 or 15 randomly. There is possibility
> that the value may be 1. The may problem here is the overflow of addition of
> Long type. You can reproduce the overflow problem by following code
> {code}
> public static void main(String[] args) {
> long a1= 9223372036854775807L;
> long a2=1022672;
> long res = a1+a2;
> System.out.println(res); //-9223372036853753137
> BigInteger b1= BigInteger.valueOf(a1);
> BigInteger b2 = BigInteger.valueOf(a2);
> BigInteger bigRes = b1.add(b2);
> System.out.println(bigRes); //9223372036855798479
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17010) Fix the overflow problem of Long type in SetSparkReducerParallelism
[
https://issues.apache.org/jira/browse/HIVE-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077534#comment-16077534
]
Rui Li edited comment on HIVE-17010 at 7/7/17 3:53 AM:
---
+1
was (Author: lirui):
1
> Fix the overflow problem of Long type in SetSparkReducerParallelism
> ---
>
> Key: HIVE-17010
> URL: https://issues.apache.org/jira/browse/HIVE-17010
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-17010.1.patch, HIVE-17010.2.patch,
> HIVE-17010.3.patch
>
>
> We use
> [numberOfByteshttps://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L129]
> to collect the numberOfBytes of sibling of specified RS. We use Long type
> and it happens overflow when the data is too big. After happening this
> situation, the parallelism is decided by
> [sparkMemoryAndCores.getSecond()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L184]
> if spark.dynamic.allocation.enabled is true, sparkMemoryAndCores.getSecond
> is a dymamic value which is decided by spark runtime. For example, the value
> of sparkMemoryAndCores.getSecond is 5 or 15 randomly. There is possibility
> that the value may be 1. The may problem here is the overflow of addition of
> Long type. You can reproduce the overflow problem by following code
> {code}
> public static void main(String[] args) {
> long a1= 9223372036854775807L;
> long a2=1022672;
> long res = a1+a2;
> System.out.println(res); //-9223372036853753137
> BigInteger b1= BigInteger.valueOf(a1);
> BigInteger b2 = BigInteger.valueOf(a2);
> BigInteger bigRes = b1.add(b2);
> System.out.println(bigRes); //9223372036855798479
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17010) Fix the overflow problem of Long type in SetSparkReducerParallelism
[
https://issues.apache.org/jira/browse/HIVE-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077557#comment-16077557
]
Hive QA commented on HIVE-17010:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12876022/HIVE-17010.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10834 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=232)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=177)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5914/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5914/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5914/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12876022 - PreCommit-HIVE-Build
> Fix the overflow problem of Long type in SetSparkReducerParallelism
> ---
>
> Key: HIVE-17010
> URL: https://issues.apache.org/jira/browse/HIVE-17010
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-17010.1.patch, HIVE-17010.2.patch,
> HIVE-17010.3.patch
>
>
> We use
> [numberOfByteshttps://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L129]
> to collect the numberOfBytes of sibling of specified RS. We use Long type
> and it happens overflow when the data is too big. After happening this
> situation, the parallelism is decided by
> [sparkMemoryAndCores.getSecond()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L184]
> if spark.dynamic.allocation.enabled is true, sparkMemoryAndCores.getSecond
> is a dymamic value which is decided by spark runtime. For example, the value
> of sparkMemoryAndCores.getSecond is 5 or 15 randomly. There is possibility
> that the value may be 1. The may problem here is the overflow of addition of
> Long type. You can reproduce the overflow problem by following code
> {code}
> public static void main(String[] args) {
> long a1= 9223372036854775807L;
> long a2=1022672;
> long res = a1+a2;
> System.out.println(res); //-9223372036853753137
> BigInteger b1= BigInteger.valueOf(a1);
> BigInteger b2 = BigInteger.valueOf(a2);
> BigInteger bigRes = b1.add(b2);
> System.out.println(bigRes); //9223372036855798479
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17051) Each table metadata is requested twice during query compile
[
https://issues.apache.org/jira/browse/HIVE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077571#comment-16077571
]
Rajesh Balamohan commented on HIVE-17051:
-
I believe {{CalcitePlanner::genOPTree --> init}} might be resetting the cache,
which is forcing the table lookup again during compilation phase.
> Each table metadata is requested twice during query compile
> ---
>
> Key: HIVE-17051
> URL: https://issues.apache.org/jira/browse/HIVE-17051
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Labels: performance
>
> As far as I can tell, for each table referenced in a query the metadata is
> retrieved twice during compilation:
> first call:
> {noformat}
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1320)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1275)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getTableObjectByName(SemanticAnalyzer.java:10943)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1992)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1942)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:11178)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11309)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:295)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:261)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:566)
> {noformat}
> second call:
> {noformat}
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1320)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1275)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getTableObjectByName(SemanticAnalyzer.java:10943)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1992)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1942)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1934)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:431)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11320)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:295)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:261)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:566)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-13882) When hive.server2.async.exec.async.compile is turned on, from JDBC we will get "The query did not generate a result set"
[
https://issues.apache.org/jira/browse/HIVE-13882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313448#comment-15313448
]
Lefty Leverenz edited comment on HIVE-13882 at 7/7/17 5:57 AM:
---
Doc note: This changes the description of *hive.driver.parallel.compilation*,
which was introduced by HIVE-4239 in 2.0.0. It needs to be documented in the
HiveServer2 section of Configuration Properties.
* [Configuration Properties -- HiveServer2 |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-HiveServer2]
Added a TODOC2.1 label.
Edit 06/Jul/17: Now it's a TODOC2.2 label. The patch was reverted from
branch-2.1 in June 2016.
was (Author: le...@hortonworks.com):
Doc note: This changes the description of *hive.driver.parallel.compilation*,
which was introduced by HIVE-4239 in 2.0.0. It needs to be documented in the
HiveServer2 section of Configuration Properties.
* [Configuration Properties -- HiveServer2 |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-HiveServer2]
Added a TODOC2.1 label.
> When hive.server2.async.exec.async.compile is turned on, from JDBC we will
> get "The query did not generate a result set"
> -
>
> Key: HIVE-13882
> URL: https://issues.apache.org/jira/browse/HIVE-13882
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-13882.1.patch, HIVE-13882.2.patch
>
>
> The following would fail with "The query did not generate a result set"
> stmt.execute("SET hive.driver.parallel.compilation=true");
> stmt.execute("SET hive.server2.async.exec.async.compile=true");
> ResultSet res = stmt.executeQuery("SELECT * FROM " + tableName);
> res.next();
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313453#comment-15313453 ] Lefty Leverenz edited comment on HIVE-4239 at 7/7/17 5:58 AM: -- HIVE-13882 changes the description of *hive.driver.parallel.compilation* in release 2.1.0. Edit 06/Jul/17: Actually that's in release 2.2.0, because the commit to 2.1.0 was reverted. was (Author: le...@hortonworks.com): HIVE-13882 changes the description of *hive.driver.parallel.compilation* in release 2.1.0. > Remove lock on compilation stage > > > Key: HIVE-4239 > URL: https://issues.apache.org/jira/browse/HIVE-4239 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Query Processor >Reporter: Carl Steinbach >Assignee: Sergey Shelukhin > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, > HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, > HIVE-4239.06.patch, HIVE-4239.07.patch, HIVE-4239.08.patch, HIVE-4239.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
