[jira] [Updated] (HIVE-12412) Multi insert queries fail to run properly in hive 1.1.x or later.
[
https://issues.apache.org/jira/browse/HIVE-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-12412:
Affects Version/s: 2.3.0
> Multi insert queries fail to run properly in hive 1.1.x or later.
> -
>
> Key: HIVE-12412
> URL: https://issues.apache.org/jira/browse/HIVE-12412
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.0, 2.3.0
>Reporter: John P. Petrakis
> Labels: Correctness, CorrectnessBug
>
> We use multi insert queries to take data in one table and manipulate it by
> inserting it into a results table. Queries are of this form:
> from (select * from data_table lateral view explode(data_table.f2) f2 as
> explode_f2) as explode_data_table
>insert overwrite table results_table partition (q_id='C.P1',rl='1')
>select
>array(cast(if(explode_data_table.f1 is null or
> explode_data_table.f1='', 'UNKNOWN',explode_data_table.f1) as
> String),cast(explode_f2.s1 as String)) as dimensions,
>ARRAY(CAST(sum(explode_f2.d1) as Double)) as metrics,
>null as rownm
>where (explode_data_table.date_id between 20151016 and 20151016)
>group by
>if(explode_data_table.f1 is null or explode_data_table.f1='',
> 'UNKNOWN',explode_data_table.f1),
>explode_f2.s1
>INSERT OVERWRITE TABLE results_table PARTITION (q_id='C.P2',rl='0')
>SELECT ARRAY(CAST('Total' as String),CAST('Total' as String)) AS
> dimensions,
>ARRAY(CAST(sum(explode_f2.d1) as Double)) AS metrics,
>null AS rownm
>WHERE (explode_data_table.date_id BETWEEN 20151016 AND 20151016)
>INSERT OVERWRITE TABLE results_table PARTITION (q_id='C.P5',rl='0')
>SELECT
>ARRAY(CAST('Total' as String)) AS dimensions,
>ARRAY(CAST(sum(explode_f2.d1) as Double)) AS metrics,
>null AS rownm
>WHERE (explode_data_table.date_id BETWEEN 20151016 AND 20151016)
> This query is meant to total a given field of a struct that is potentially a
> list of structs. For our test data set, which consists of a single row, the
> summation yields "Null", with messages in the hive log of the nature:
> Missing fields! Expected 2 fields but only got 1! Ignoring similar problems.
> or "Extra fields detected..."
> For significantly more data, this query will eventually cause a run time
> error while processing a column (caused by array index out of bounds
> exception in one of the lazy binary classes such as LazyBinaryString or
> LazyBinaryStruct).
> Using the query above from the hive command line, the following data was used:
> (note there are tabs in the data below)
> string oneone:1.0:1.00:10.0,eon:1.0:1.00:100.0
> string twotwo:2.0:2.00:20.0,otw:2.0:2.00:20.0,wott:2.0:2.00:20.0
> string thrthree:3.0:3.00:30.0
> string foufour:4.0:4.00:40.0
> There are two fields, a string, (eg. 'string one') and a list of structs.
> The following is used to create the table:
> create table if not exists t1 (
> f1 string,
> f2
> array>
> )
> partitioned by (clid string, date_id string)
> row format delimited fields
> terminated by '09'
> collection items terminated by ','
> map keys terminated by ':'
> lines terminated by '10'
> location '/user/hive/warehouse/t1';
> And the following is used to load the data:
> load data local inpath '/path/to/data/file/cplx_test.data2' OVERWRITE into
> table t1 partition(client_id='987654321',date_id='20151016');
> The resulting table should yield the following:
> ["string fou","four"] [4.0] nullC.P11
> ["string one","eon"] [1.0] nullC.P11
> ["string one","one"] [1.0] nullC.P11
> ["string thr","three"][3.0] nullC.P11
> ["string two","otw"] [2.0] nullC.P11
> ["string two","two"] [2.0] nullC.P11
> ["string two","wott"] [2.0] nullC.P11
> ["Total","Total"] [15.0] nullC.P20
> ["Total"] [15.0] nullC.P50
> However what we get is:
> Hive Runtime Error while processing row
> {"_col2":2.5306499719322744E-258,"_col3":""} (ultimately due to an array
> index out of bounds exception)
> If we reduce the above data to a SINGLE row, the we don't get an exception
> but the total fields come out as NULL.
> The ONLY way this query would work is
> 1) if I added a group by (date_id) or even group by ('') as the last line in
> the query... or removed the last where clause for the final insert. (The
> reason why w
[jira] [Updated] (HIVE-16413) Create table as select does not check ownership of the location
[
https://issues.apache.org/jira/browse/HIVE-16413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-16413:
Attachment: HIVE-16413.patch
> Create table as select does not check ownership of the location
> ---
>
> Key: HIVE-16413
> URL: https://issues.apache.org/jira/browse/HIVE-16413
> Project: Hive
> Issue Type: Bug
> Components: Authorization, SQLStandardAuthorization
>Affects Versions: 1.3.0, 1.2.2, 2.1.1
> Environment: hive-1.2.2, with following conf:
> hive.security.authorization.enabled: true
> hive.security.authorization.manager:
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory
> hive.security.authenticator.manager:
> org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Fix For: 2.2.0
>
> Attachments: HIVE-16413.patch
>
>
> 1. following statement failed:
> {code}
> create table foo(id int) location 'hdfs:///tmp/foo';
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: Principal [name=userx, type=USER] does not have following
> privileges for operation CREATETABLE [[OBJECT OWNERSHIP] on Object
> [type=DFS_URI, name=hdfs://hacluster/tmp/foo]] (state=42000,code=4)
> {code}
> 2. but when use create table as select, it successed:
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> create table foo location
> 'hdfs:///tmp/foo' as select * from xxx2;
> INFO : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO : number of splits:1
> INFO : Submitting tokens for job: job_1491449632882_0094
> INFO : Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:hacluster
> INFO : The url to track the job:
> https://189-39-151-44:26001/proxy/application_1491449632882_0094/
> INFO : Starting Job = job_1491449632882_0094, Tracking URL =
> https://189-39-151-44:26001/proxy/application_1491449632882_0094/
> INFO : Kill Command = /opt/hive-1.3.0/bin/..//../hadoop/bin/hadoop job
> -kill job_1491449632882_0094
> INFO : Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> INFO : 2017-04-10 09:44:49,185 Stage-1 map = 0%, reduce = 0%
> INFO : 2017-04-10 09:44:57,202 Stage-1 map = 100%, reduce = 0%, Cumulative
> CPU 1.98 sec
> INFO : MapReduce Total cumulative CPU time: 1 seconds 980 msec
> INFO : Ended Job = job_1491449632882_0094
> INFO : Stage-3 is selected by condition resolver.
> INFO : Stage-2 is filtered out by condition resolver.
> INFO : Stage-4 is filtered out by condition resolver.
> INFO : Moving data to directory
> hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10001
> from
> hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10003
> INFO : Moving data to directory hdfs:/tmp/foo from
> hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10001
> No rows affected (26.969 seconds)
> {code}
> 3. and the table location is hdfs://hacluster/tmp/foo :
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> desc formatted foo;
> +---+---+---+--+
> | col_name| data_type
> |comment|
> +---+---+---+--+
> | # col_name| data_type
> | comment |
> | | NULL
> | NULL |
> | id| int
> | |
> | | NULL
> | NULL |
> | # Detailed Table Information | NULL
> | NULL |
> | Database: | default
> | NULL |
> | Owner:| userx
> | NULL |
> | CreateTime: | Mon Apr 10 09:44:59 CST 2017
> | NULL |
> | LastAccessTime: | UNKNOWN
> | NULL |
> | Protect Mode: | None
> | NULL |
> | Retention:
[jira] [Updated] (HIVE-16413) Create table as select does not check ownership of the location
[
https://issues.apache.org/jira/browse/HIVE-16413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-16413:
Fix Version/s: 2.2.0
Status: Patch Available (was: Open)
> Create table as select does not check ownership of the location
> ---
>
> Key: HIVE-16413
> URL: https://issues.apache.org/jira/browse/HIVE-16413
> Project: Hive
> Issue Type: Bug
> Components: Authorization, SQLStandardAuthorization
>Affects Versions: 2.1.1, 1.2.2, 1.3.0
> Environment: hive-1.2.2, with following conf:
> hive.security.authorization.enabled: true
> hive.security.authorization.manager:
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory
> hive.security.authenticator.manager:
> org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Fix For: 2.2.0
>
> Attachments: HIVE-16413.patch
>
>
> 1. following statement failed:
> {code}
> create table foo(id int) location 'hdfs:///tmp/foo';
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: Principal [name=userx, type=USER] does not have following
> privileges for operation CREATETABLE [[OBJECT OWNERSHIP] on Object
> [type=DFS_URI, name=hdfs://hacluster/tmp/foo]] (state=42000,code=4)
> {code}
> 2. but when use create table as select, it successed:
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> create table foo location
> 'hdfs:///tmp/foo' as select * from xxx2;
> INFO : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO : number of splits:1
> INFO : Submitting tokens for job: job_1491449632882_0094
> INFO : Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:hacluster
> INFO : The url to track the job:
> https://189-39-151-44:26001/proxy/application_1491449632882_0094/
> INFO : Starting Job = job_1491449632882_0094, Tracking URL =
> https://189-39-151-44:26001/proxy/application_1491449632882_0094/
> INFO : Kill Command = /opt/hive-1.3.0/bin/..//../hadoop/bin/hadoop job
> -kill job_1491449632882_0094
> INFO : Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> INFO : 2017-04-10 09:44:49,185 Stage-1 map = 0%, reduce = 0%
> INFO : 2017-04-10 09:44:57,202 Stage-1 map = 100%, reduce = 0%, Cumulative
> CPU 1.98 sec
> INFO : MapReduce Total cumulative CPU time: 1 seconds 980 msec
> INFO : Ended Job = job_1491449632882_0094
> INFO : Stage-3 is selected by condition resolver.
> INFO : Stage-2 is filtered out by condition resolver.
> INFO : Stage-4 is filtered out by condition resolver.
> INFO : Moving data to directory
> hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10001
> from
> hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10003
> INFO : Moving data to directory hdfs:/tmp/foo from
> hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10001
> No rows affected (26.969 seconds)
> {code}
> 3. and the table location is hdfs://hacluster/tmp/foo :
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> desc formatted foo;
> +---+---+---+--+
> | col_name| data_type
> |comment|
> +---+---+---+--+
> | # col_name| data_type
> | comment |
> | | NULL
> | NULL |
> | id| int
> | |
> | | NULL
> | NULL |
> | # Detailed Table Information | NULL
> | NULL |
> | Database: | default
> | NULL |
> | Owner:| userx
> | NULL |
> | CreateTime: | Mon Apr 10 09:44:59 CST 2017
> | NULL |
> | LastAccessTime: | UNKNOWN
> | NULL |
> | Protect Mode: | None
> | NULL
[jira] [Updated] (HIVE-16413) Create table as select does not check ownership of the location
[
https://issues.apache.org/jira/browse/HIVE-16413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-16413:
Description:
1. following statement failed:
{code}
create table foo(id int) location 'hdfs:///tmp/foo';
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: Principal [name=userx, type=USER] does not have following
privileges for operation CREATETABLE [[OBJECT OWNERSHIP] on Object
[type=DFS_URI, name=hdfs://hacluster/tmp/foo]] (state=42000,code=4)
{code}
2. but when use create table as select, it successed:
{code}
0: jdbc:hive2://189.39.151.44:21066/> create table foo location
'hdfs:///tmp/foo' as select * from xxx2;
INFO : Number of reduce tasks is set to 0 since there's no reduce operator
INFO : number of splits:1
INFO : Submitting tokens for job: job_1491449632882_0094
INFO : Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:hacluster
INFO : The url to track the job:
https://189-39-151-44:26001/proxy/application_1491449632882_0094/
INFO : Starting Job = job_1491449632882_0094, Tracking URL =
https://189-39-151-44:26001/proxy/application_1491449632882_0094/
INFO : Kill Command = /opt/hive-1.3.0/bin/..//../hadoop/bin/hadoop job -kill
job_1491449632882_0094
INFO : Hadoop job information for Stage-1: number of mappers: 1; number of
reducers: 0
INFO : 2017-04-10 09:44:49,185 Stage-1 map = 0%, reduce = 0%
INFO : 2017-04-10 09:44:57,202 Stage-1 map = 100%, reduce = 0%, Cumulative
CPU 1.98 sec
INFO : MapReduce Total cumulative CPU time: 1 seconds 980 msec
INFO : Ended Job = job_1491449632882_0094
INFO : Stage-3 is selected by condition resolver.
INFO : Stage-2 is filtered out by condition resolver.
INFO : Stage-4 is filtered out by condition resolver.
INFO : Moving data to directory
hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10001
from
hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10003
INFO : Moving data to directory hdfs:/tmp/foo from
hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10001
No rows affected (26.969 seconds)
{code}
3. and the table location is hdfs://hacluster/tmp/foo :
{code}
0: jdbc:hive2://189.39.151.44:21066/> desc formatted foo;
+---+---+---+--+
| col_name| data_type
|comment|
+---+---+---+--+
| # col_name| data_type
| comment |
| | NULL
| NULL |
| id| int
| |
| | NULL
| NULL |
| # Detailed Table Information | NULL
| NULL |
| Database: | default
| NULL |
| Owner:| userx
| NULL |
| CreateTime: | Mon Apr 10 09:44:59 CST 2017
| NULL |
| LastAccessTime: | UNKNOWN
| NULL |
| Protect Mode: | None
| NULL |
| Retention:| 0
| NULL |
| Location: | hdfs://hacluster/tmp/foo
| NULL |
| Table Type: | MANAGED_TABLE
| NULL |
| Table Parameters: | NULL
| NULL |
| | COLUMN_STATS_ACCURATE
| false |
| | numFiles
| 1 |
| | numRows
| -1|
| | rawDataSize
| -1|
| | totalSize
| 56|
|
[jira] [Assigned] (HIVE-16413) Create table as select does not check ownership of the location
[
https://issues.apache.org/jira/browse/HIVE-16413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao reassigned HIVE-16413:
---
> Create table as select does not check ownership of the location
> ---
>
> Key: HIVE-16413
> URL: https://issues.apache.org/jira/browse/HIVE-16413
> Project: Hive
> Issue Type: Bug
> Components: Authorization, SQLStandardAuthorization
>Affects Versions: 2.1.1, 1.2.2, 1.3.0
> Environment: hive-1.2.2, with following conf:
> hive.security.authorization.enabled: true
> hive.security.authorization.manager:
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory
> hive.security.authenticator.manager:
> org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>
> 1. following statement failed:
> {code}
> create table foo(id int) location 'hdfs:///tmp/foo';
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: Principal [name=userx, type=USER] does not have following
> privileges for operation CREATETABLE [[OBJECT OWNERSHIP] on Object
> [type=DFS_URI, name=hdfs://hacluster/tmp/foo]] (state=42000,code=4)
> {code}
> 2. but when use create table as select, it successed:
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> create table foo location
> 'hdfs:///tmp/foo' as select * from xxx2;
> INFO : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO : number of splits:1
> INFO : Submitting tokens for job: job_1491449632882_0094
> INFO : Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:hacluster
> INFO : The url to track the job:
> https://189-39-151-44:26001/proxy/application_1491449632882_0094/
> INFO : Starting Job = job_1491449632882_0094, Tracking URL =
> https://189-39-151-44:26001/proxy/application_1491449632882_0094/
> INFO : Kill Command =
> /opt/huawei/Bigdata/FusionInsight_V100R002C60U20/FusionInsight-Hive-1.3.0/hive-1.3.0/bin/..//../hadoop/bin/hadoop
> job -kill job_1491449632882_0094
> INFO : Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> INFO : 2017-04-10 09:44:49,185 Stage-1 map = 0%, reduce = 0%
> INFO : 2017-04-10 09:44:57,202 Stage-1 map = 100%, reduce = 0%, Cumulative
> CPU 1.98 sec
> INFO : MapReduce Total cumulative CPU time: 1 seconds 980 msec
> INFO : Ended Job = job_1491449632882_0094
> INFO : Stage-3 is selected by condition resolver.
> INFO : Stage-2 is filtered out by condition resolver.
> INFO : Stage-4 is filtered out by condition resolver.
> INFO : Moving data to directory
> hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10001
> from
> hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10003
> INFO : Moving data to directory hdfs:/tmp/foo from
> hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10001
> No rows affected (26.969 seconds)
> {code}
> 3. and the table location is hdfs://hacluster/tmp/foo :
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> desc formatted foo;
> +---+---+---+--+
> | col_name| data_type
> |comment|
> +---+---+---+--+
> | # col_name| data_type
> | comment |
> | | NULL
> | NULL |
> | id| int
> | |
> | | NULL
> | NULL |
> | # Detailed Table Information | NULL
> | NULL |
> | Database: | default
> | NULL |
> | Owner:| userx
> | NULL |
> | CreateTime: | Mon Apr 10 09:44:59 CST 2017
> | NULL |
> | LastAccessTime: | UNKNOWN
> | NULL |
> | Protect Mode: | None
> | NULL |
> | Retention:| 0
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923852#comment-15923852
]
Niklaus Xiao commented on HIVE-15956:
-
[~kgyrtkirk] This patch works for me, maybe the default batch size is too large
for you, you can change {{hive.metastore.drop.partitions.batch.size}} to 200
and try again.
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897176#comment-15897176
]
Niklaus Xiao commented on HIVE-15956:
-
Tests failure not related.
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-15956:
Attachment: HIVE-15956.patch
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao reassigned HIVE-15956:
---
Assignee: Niklaus Xiao
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15896962#comment-15896962
]
Niklaus Xiao commented on HIVE-15956:
-
Add logic to drop partitions in batches, default batch size is 1000.
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-15956:
Status: Patch Available (was: Open)
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15666) Select query with view adds base table partition as direct input in spark engine
[
https://issues.apache.org/jira/browse/HIVE-15666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15884511#comment-15884511
]
Niklaus Xiao commented on HIVE-15666:
-
Any progress on this ?
> Select query with view adds base table partition as direct input in spark
> engine
>
>
> Key: HIVE-15666
> URL: https://issues.apache.org/jira/browse/HIVE-15666
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.3.0
>Reporter: Niklaus Xiao
>Assignee: Aihua Xu
> Attachments: TestViewEntityInSparkEngine.patch
>
>
> repro steps:
> {code}
> set hive.execution.engine=spark;
> create table base(id int) partitioned by (dt string);
> alter table base add partition(dt='2017');
> create view view1 as select * from base where id < 10;
> select * from view1;
> {code}
> it requires the access not only for view1 but also for base@dt=2017
> partition, which should not be required.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15666) Select query with view adds base table partition as direct input in spark engine
[
https://issues.apache.org/jira/browse/HIVE-15666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15853435#comment-15853435
]
Niklaus Xiao commented on HIVE-15666:
-
[~aihuaxu] this is my test result with the latest version, 2.2.0-SNAPSHOT:
MR + CBO on -> issue exists
MR + CBO off -> issue doesnot exists
Spark + CBO on -> issue exists
> Select query with view adds base table partition as direct input in spark
> engine
>
>
> Key: HIVE-15666
> URL: https://issues.apache.org/jira/browse/HIVE-15666
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.3.0
>Reporter: Niklaus Xiao
>Assignee: Aihua Xu
> Attachments: TestViewEntityInSparkEngine.patch
>
>
> repro steps:
> {code}
> set hive.execution.engine=spark;
> create table base(id int) partitioned by (dt string);
> alter table base add partition(dt='2017');
> create view view1 as select * from base where id < 10;
> select * from view1;
> {code}
> it requires the access not only for view1 but also for base@dt=2017
> partition, which should not be required.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15666) Select query with view adds base table partition as direct input in spark engine
[
https://issues.apache.org/jira/browse/HIVE-15666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15841905#comment-15841905
]
Niklaus Xiao commented on HIVE-15666:
-
[~aihuaxu] I tried 1.3.0-SNAPSHOT didn't see the issue when use MR engine, I'll
try to use the latest version.
> Select query with view adds base table partition as direct input in spark
> engine
>
>
> Key: HIVE-15666
> URL: https://issues.apache.org/jira/browse/HIVE-15666
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.3.0
>Reporter: Niklaus Xiao
>Assignee: Aihua Xu
> Attachments: TestViewEntityInSparkEngine.patch
>
>
> repro steps:
> {code}
> set hive.execution.engine=spark;
> create table base(id int) partitioned by (dt string);
> alter table base add partition(dt='2017');
> create view view1 as select * from base where id < 10;
> select * from view1;
> {code}
> it requires the access not only for view1 but also for base@dt=2017
> partition, which should not be required.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15193) Drop non-existing view executes successfully
[
https://issues.apache.org/jira/browse/HIVE-15193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831044#comment-15831044
]
Niklaus Xiao commented on HIVE-15193:
-
set {{hive.exec.drop.ignorenonexistent}} to false and try again.
{code}
0: jdbc:hive2://189.39.151.74:21066/> drop view aaa;
No rows affected (0.113 seconds)
0: jdbc:hive2://189.39.151.74:21066/> set
hive.exec.drop.ignorenonexistent=false;
No rows affected (0.009 seconds)
0: jdbc:hive2://189.39.151.74:21066/> drop view aaa;
Error: Error while compiling statement: FAILED: SemanticException [Error
10001]: Table not found aaa (state=42S02,code=10001)
{code}
> Drop non-existing view executes successfully
>
>
> Key: HIVE-15193
> URL: https://issues.apache.org/jira/browse/HIVE-15193
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema
>Reporter: Sangeeta Gulia
>
> When i try to drop a view which do not exist. Then the query result gives me
> Output: Ok
> Whereas it should give an error that "The view you are trying to drop does
> not exist"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15666) Select query with view adds base table partition as direct input in spark engine
[
https://issues.apache.org/jira/browse/HIVE-15666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15830909#comment-15830909
]
Niklaus Xiao commented on HIVE-15666:
-
Yes, MR engine works fine.
> Select query with view adds base table partition as direct input in spark
> engine
>
>
> Key: HIVE-15666
> URL: https://issues.apache.org/jira/browse/HIVE-15666
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.3.0
>Reporter: Niklaus Xiao
>Assignee: Aihua Xu
> Attachments: TestViewEntityInSparkEngine.patch
>
>
> repro steps:
> {code}
> set hive.execution.engine=spark;
> create table base(id int) partitioned by (dt string);
> alter table base add partition(dt='2017');
> create view view1 as select * from base where id < 10;
> select * from view1;
> {code}
> it requires the access not only for view1 but also for base@dt=2017
> partition, which should not be required.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15666) Select query with view adds base table partition as direct input in spark engine
[
https://issues.apache.org/jira/browse/HIVE-15666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15829892#comment-15829892
]
Niklaus Xiao commented on HIVE-15666:
-
Add a test case to reproduce the issue.
cc [~aihuaxu] & [~navis] similar issue
https://issues.apache.org/jira/browse/HIVE-14805,
https://issues.apache.org/jira/browse/HIVE-10875
> Select query with view adds base table partition as direct input in spark
> engine
>
>
> Key: HIVE-15666
> URL: https://issues.apache.org/jira/browse/HIVE-15666
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.3.0
>Reporter: Niklaus Xiao
> Attachments: TestViewEntityInSparkEngine.patch
>
>
> repro steps:
> {code}
> set hive.execution.engine=spark;
> create table base(id int) partitioned by (dt string);
> alter table base add partition(dt='2017');
> create view view1 as select * from base where id < 10;
> select * from view1;
> {code}
> it requires the access not only for view1 but also for base@dt=2017
> partition, which should not be required.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15666) Select query with view adds base table partition as direct input in spark engine
[
https://issues.apache.org/jira/browse/HIVE-15666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-15666:
Description:
repro steps:
{code}
set hive.execution.engine=spark;
create table base(id int) partitioned by (dt string);
alter table base add partition(dt='2017');
create view view1 as select * from base where id < 10;
select * from view1;
{code}
it requires the access not only for view1 but also for base@dt=2017 partition,
which should not be required.
was:
repo steps:
{code}
set hive.execution.engine=spark;
create table base(id int) partitioned by (dt string);
alter table base add partition(dt='2017');
create view view1 as select * from base where id < 10;
select * from view1;
{code}
it requires the access not only for view1 but also for base@dt=2017 partition,
which should not be required.
> Select query with view adds base table partition as direct input in spark
> engine
>
>
> Key: HIVE-15666
> URL: https://issues.apache.org/jira/browse/HIVE-15666
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.3.0
>Reporter: Niklaus Xiao
> Attachments: TestViewEntityInSparkEngine.patch
>
>
> repro steps:
> {code}
> set hive.execution.engine=spark;
> create table base(id int) partitioned by (dt string);
> alter table base add partition(dt='2017');
> create view view1 as select * from base where id < 10;
> select * from view1;
> {code}
> it requires the access not only for view1 but also for base@dt=2017
> partition, which should not be required.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15666) Select query with view adds base table partition as direct input in spark engine
[
https://issues.apache.org/jira/browse/HIVE-15666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-15666:
Attachment: TestViewEntityInSparkEngine.patch
> Select query with view adds base table partition as direct input in spark
> engine
>
>
> Key: HIVE-15666
> URL: https://issues.apache.org/jira/browse/HIVE-15666
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.3.0
>Reporter: Niklaus Xiao
> Attachments: TestViewEntityInSparkEngine.patch
>
>
> repo steps:
> {code}
> set hive.execution.engine=spark;
> create table base(id int) partitioned by (dt string);
> alter table base add partition(dt='2017');
> create view view1 as select * from base where id < 10;
> select * from view1;
> {code}
> it requires the access not only for view1 but also for base@dt=2017
> partition, which should not be required.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15483) Database and table name is case sensitive when used in show grant
[
https://issues.apache.org/jira/browse/HIVE-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15774902#comment-15774902
]
Niklaus Xiao commented on HIVE-15483:
-
cc [~navis]
> Database and table name is case sensitive when used in show grant
> -
>
> Key: HIVE-15483
> URL: https://issues.apache.org/jira/browse/HIVE-15483
> Project: Hive
> Issue Type: Bug
> Components: Authorization, SQLStandardAuthorization
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15483.1.patch
>
>
> When use SQLStdAuth, db name and table name is case sensitive when use
> {{show grant}} command.
> {code}
> 0: jdbc:hive2://localhost:21066/> show grant on table p1;
> +---+++-+--+-++---++--+--+
> | database | table | partition | column |principal_name|
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+--+-++---++--+--+
> | default | p1 || | userx| USER
>| DELETE | true | 1481872357000 | userx|
> | default | p1 || | userx| USER
>| INSERT | true | 1481872357000 | userx|
> | default | p1 || | userx| USER
>| SELECT | true | 1481872357000 | userx|
> | default | p1 || | userx| USER
>| UPDATE | true | 1481872357000 | userx|
> +---+++-+--+-++---++--+--+
> 7 rows selected (0.158 seconds)
> 0: jdbc:hive2://localhost:21066/> show grant on table P1;
> +---+++-+-+-++---+-+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+-+--+--+
> +---+++-+-+-++---+-+--+--+
> No rows selected (9.608 seconds)
> 0: jdbc:hive2://localhost:21066/> show grant on table defaulT.p1;
> +---+++-+-+-++---+-+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+-+--+--+
> +---+++-+-+-++---+-+--+--+
> No rows selected (0.06 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15483) Database and table name is case sensitive when used in show grant
[
https://issues.apache.org/jira/browse/HIVE-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15766882#comment-15766882
]
Niklaus Xiao commented on HIVE-15483:
-
Test failures not related. [~ashutoshc] can you take a look?
> Database and table name is case sensitive when used in show grant
> -
>
> Key: HIVE-15483
> URL: https://issues.apache.org/jira/browse/HIVE-15483
> Project: Hive
> Issue Type: Bug
> Components: Authorization, SQLStandardAuthorization
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15483.1.patch
>
>
> When use SQLStdAuth, db name and table name is case sensitive when use
> {{show grant}} command.
> {code}
> 0: jdbc:hive2://localhost:21066/> show grant on table p1;
> +---+++-+--+-++---++--+--+
> | database | table | partition | column |principal_name|
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+--+-++---++--+--+
> | default | p1 || | userx| USER
>| DELETE | true | 1481872357000 | userx|
> | default | p1 || | userx| USER
>| INSERT | true | 1481872357000 | userx|
> | default | p1 || | userx| USER
>| SELECT | true | 1481872357000 | userx|
> | default | p1 || | userx| USER
>| UPDATE | true | 1481872357000 | userx|
> +---+++-+--+-++---++--+--+
> 7 rows selected (0.158 seconds)
> 0: jdbc:hive2://localhost:21066/> show grant on table P1;
> +---+++-+-+-++---+-+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+-+--+--+
> +---+++-+-+-++---+-+--+--+
> No rows selected (9.608 seconds)
> 0: jdbc:hive2://localhost:21066/> show grant on table defaulT.p1;
> +---+++-+-+-++---+-+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+-+--+--+
> +---+++-+-+-++---+-+--+--+
> No rows selected (0.06 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15483) Database and table name is case sensitive when used in show grant
[
https://issues.apache.org/jira/browse/HIVE-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-15483:
Attachment: HIVE-15483.1.patch
> Database and table name is case sensitive when used in show grant
> -
>
> Key: HIVE-15483
> URL: https://issues.apache.org/jira/browse/HIVE-15483
> Project: Hive
> Issue Type: Bug
> Components: Authorization, SQLStandardAuthorization
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15483.1.patch
>
>
> When use SQLStdAuth, db name and table name is case sensitive when use
> {{show grant}} command.
> {code}
> 0: jdbc:hive2://localhost:21066/> show grant on table p1;
> +---+++-+--+-++---++--+--+
> | database | table | partition | column |principal_name|
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+--+-++---++--+--+
> | default | p1 || | userx| USER
>| DELETE | true | 1481872357000 | userx|
> | default | p1 || | userx| USER
>| INSERT | true | 1481872357000 | userx|
> | default | p1 || | userx| USER
>| SELECT | true | 1481872357000 | userx|
> | default | p1 || | userx| USER
>| UPDATE | true | 1481872357000 | userx|
> +---+++-+--+-++---++--+--+
> 7 rows selected (0.158 seconds)
> 0: jdbc:hive2://localhost:21066/> show grant on table P1;
> +---+++-+-+-++---+-+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+-+--+--+
> +---+++-+-+-++---+-+--+--+
> No rows selected (9.608 seconds)
> 0: jdbc:hive2://localhost:21066/> show grant on table defaulT.p1;
> +---+++-+-+-++---+-+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+-+--+--+
> +---+++-+-+-++---+-+--+--+
> No rows selected (0.06 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15483) Database and table name is case sensitive when used in show grant
[
https://issues.apache.org/jira/browse/HIVE-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-15483:
Fix Version/s: 2.2.0
Status: Patch Available (was: Open)
> Database and table name is case sensitive when used in show grant
> -
>
> Key: HIVE-15483
> URL: https://issues.apache.org/jira/browse/HIVE-15483
> Project: Hive
> Issue Type: Bug
> Components: Authorization, SQLStandardAuthorization
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15483.1.patch
>
>
> When use SQLStdAuth, db name and table name is case sensitive when use
> {{show grant}} command.
> {code}
> 0: jdbc:hive2://localhost:21066/> show grant on table p1;
> +---+++-+--+-++---++--+--+
> | database | table | partition | column |principal_name|
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+--+-++---++--+--+
> | default | p1 || | userx| USER
>| DELETE | true | 1481872357000 | userx|
> | default | p1 || | userx| USER
>| INSERT | true | 1481872357000 | userx|
> | default | p1 || | userx| USER
>| SELECT | true | 1481872357000 | userx|
> | default | p1 || | userx| USER
>| UPDATE | true | 1481872357000 | userx|
> +---+++-+--+-++---++--+--+
> 7 rows selected (0.158 seconds)
> 0: jdbc:hive2://localhost:21066/> show grant on table P1;
> +---+++-+-+-++---+-+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+-+--+--+
> +---+++-+-+-++---+-+--+--+
> No rows selected (9.608 seconds)
> 0: jdbc:hive2://localhost:21066/> show grant on table defaulT.p1;
> +---+++-+-+-++---+-+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+-+--+--+
> +---+++-+-+-++---+-+--+--+
> No rows selected (0.06 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15389) Backport HIVE-15239 to branch-1
[
https://issues.apache.org/jira/browse/HIVE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-15389:
Attachment: HIVE-15389.branch-1.1.patch
> Backport HIVE-15239 to branch-1
> ---
>
> Key: HIVE-15389
> URL: https://issues.apache.org/jira/browse/HIVE-15389
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.2.0, 2.1.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Fix For: 1.3.0
>
> Attachments: HIVE-15389.branch-1.1.patch
>
>
> env: hive on spark engine
> reproduce step:
> {code}
> create table a1(KEHHAO string, START_DT string) partitioned by (END_DT
> string);
> create table a2(KEHHAO string, START_DT string) partitioned by (END_DT
> string);
> alter table a1 add partition(END_DT='20161020');
> alter table a1 add partition(END_DT='20161021');
> insert into table a1 partition(END_DT='20161020')
> values('2000721360','20161001');
> SELECT T1.KEHHAO,COUNT(1) FROM (
> SELECT KEHHAO FROM a1 T
> WHERE T.KEHHAO = '2000721360' AND '20161018' BETWEEN T.START_DT AND
> T.END_DT-1
> UNION ALL
> SELECT KEHHAO FROM a2 T
> WHERE T.KEHHAO = '2000721360' AND '20161018' BETWEEN T.START_DT AND
> T.END_DT-1
> ) T1
> GROUP BY T1.KEHHAO
> HAVING COUNT(1)>1;
> +-+--+--+
> | t1.kehhao | _c1 |
> +-+--+--+
> | 2000721360 | 2|
> +-+--+--+
> {code}
> the result should be none record
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15389) Backport HIVE-15239 to branch-1
[
https://issues.apache.org/jira/browse/HIVE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-15389:
Fix Version/s: 1.3.0
Status: Patch Available (was: Open)
> Backport HIVE-15239 to branch-1
> ---
>
> Key: HIVE-15389
> URL: https://issues.apache.org/jira/browse/HIVE-15389
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 2.1.0, 1.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Fix For: 1.3.0
>
>
> env: hive on spark engine
> reproduce step:
> {code}
> create table a1(KEHHAO string, START_DT string) partitioned by (END_DT
> string);
> create table a2(KEHHAO string, START_DT string) partitioned by (END_DT
> string);
> alter table a1 add partition(END_DT='20161020');
> alter table a1 add partition(END_DT='20161021');
> insert into table a1 partition(END_DT='20161020')
> values('2000721360','20161001');
> SELECT T1.KEHHAO,COUNT(1) FROM (
> SELECT KEHHAO FROM a1 T
> WHERE T.KEHHAO = '2000721360' AND '20161018' BETWEEN T.START_DT AND
> T.END_DT-1
> UNION ALL
> SELECT KEHHAO FROM a2 T
> WHERE T.KEHHAO = '2000721360' AND '20161018' BETWEEN T.START_DT AND
> T.END_DT-1
> ) T1
> GROUP BY T1.KEHHAO
> HAVING COUNT(1)>1;
> +-+--+--+
> | t1.kehhao | _c1 |
> +-+--+--+
> | 2000721360 | 2|
> +-+--+--+
> {code}
> the result should be none record
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled
[
https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705113#comment-15705113
]
Niklaus Xiao commented on HIVE-14557:
-
When we have two tasks in a {{ConditionalTask}} list, say the two tasks are A
and B, and B's parent is A, {{CommonJoinResolver}} tries to convert B to a map
join task C, and replace task B with task C, but the replacement is not correct.
> Nullpointer When both SkewJoin and Mapjoin Enabled
> ---
>
> Key: HIVE-14557
> URL: https://issues.apache.org/jira/browse/HIVE-14557
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Nemon Lou
>
> The following sql failed with return code 2 on mr.
> {noformat}
> create table a(id int,id1 int);
> create table b(id int,id1 int);
> create table c(id int,id1 int);
> set hive.optimize.skewjoin=true;
> select a.id,b.id,c.id1 from a,b,c where a.id=b.id and a.id1=c.id1;
> {noformat}
> Error log as follows:
> {noformat}
> 2016-08-17 21:13:42,081 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> Id =0
>
> Id =21
>
> Id =28
>
> Id =16
>
> <\Children>
> Id = 28 null<\Parent>
> <\FS>
> <\Children>
> Id = 21 nullId = 33
> Id =33
> null
> <\Children>
> <\Parent>
> <\HASHTABLEDUMMY><\Parent>
> <\MAPJOIN>
> <\Children>
> Id = 0 null<\Parent>
> <\TS>
> <\Children>
> <\Parent>
> <\MAP>
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[21]
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Initializing dummy operator
> 2016-08-17 21:13:42,086 INFO [main]
> org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:0,
> RECORDS_IN:0,
> 2016-08-17 21:13:42,087 ERROR [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Hit error while closing
> operators - failing tree
> 2016-08-17 21:13:42,088 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Hive Runtime Error
> while closing operators
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:682)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
> ... 8 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14989) FIELDS TERMINATED BY parsing broken when delimiter is more than 1 byte
[
https://issues.apache.org/jira/browse/HIVE-14989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15584420#comment-15584420
]
Niklaus Xiao commented on HIVE-14989:
-
You should use {{MultiDelimtSerde}} in this case.
> FIELDS TERMINATED BY parsing broken when delimiter is more than 1 byte
> --
>
> Key: HIVE-14989
> URL: https://issues.apache.org/jira/browse/HIVE-14989
> Project: Hive
> Issue Type: Bug
> Components: File Formats, Parser, Reader
>Affects Versions: 0.13.0, 0.13.1
>Reporter: Ruslan Dautkhanov
>
> FIELDS TERMINATED BY parsing broken when delimiter is more than 1 byte.
> Delimiter starting from 2nd character becomes part of returned data. No
> parsed properly.
> Test case:
> {noformat}
> CREATE external TABLE test_muldelim
> ( string1 STRING,
>string2 STRING,
>string3 STRING
> )
> ROW FORMAT
>DELIMITED FIELDS TERMINATED BY '<>'
> LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> location '/user/hive/test_muldelim'
> {noformat}
> Create a text file under /user/hive/test_muldelim with following 2 lines:
> {noformat}
> data1<>data2<>data3
> aa<>bb<>cc
> {noformat}
> Now notice that two-character delimiter wasn't parsed properly:
> {noformat}
> jdbc:hive2://host.domain.com:1> select * from ruslan_test.test_muldelim ;
> ++++--+
> | test_muldelim.string1 | test_muldelim.string2 | test_muldelim.string3 |
> ++++--+
> | data1 | >data2 | >data3 |
> | aa | >bb| >cc|
> ++++--+
> 2 rows selected (0.453 seconds)
> {noformat}
> The second delimiter's character ('>') became part of the columns to the
> right (`string2` and `string3`).
> Table DDL:
> {noformat}
> 0: jdbc:hive2://host.domain.com:1> show create table dafault.test_muldelim ;
> +-+--+
> | createtab_stmt |
> +-+--+
> | CREATE EXTERNAL TABLE `default.test_muldelim`( |
> | `string1` string, |
> | `string2` string, |
> | `string3` string) |
> | ROW FORMAT DELIMITED|
> | FIELDS TERMINATED BY '<>' |
> | LINES TERMINATED BY '\n' |
> | STORED AS INPUTFORMAT |
> | 'org.apache.hadoop.mapred.TextInputFormat'|
> | OUTPUTFORMAT|
> | 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' |
> | LOCATION|
> | 'hdfs://epsdatalake/user/hive/test_muldelim' |
> | TBLPROPERTIES ( |
> | 'transient_lastDdlTime'='1476727100') |
> +-+--+
> 15 rows selected (0.286 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14867) "serialization.last.column.takes.rest" does not work for MultiDelimitSerDe
[
https://issues.apache.org/jira/browse/HIVE-14867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535660#comment-15535660
]
Niklaus Xiao commented on HIVE-14867:
-
{{LazySimpleSerDe}} works as expected:
{code}
create table t__2(a string, b string) row format delimited fields terminated by
',' stored as textfile;
{code}
load data into table t__2:
{code}
1,Lily,HW,abc
2,Lucy,LX,asdf
3,Lilei,XX,ss
{code}
select from t_2:
{code}
select * from t__2;
+-+-+--+
| t__2.a | t__2.b |
+-+-+--+
| 1 | Lily|
| 2 | Lucy|
| 3 | Lilei |
+-+-+--+
3 rows selected (0.382 seconds)
{code}
> "serialization.last.column.takes.rest" does not work for MultiDelimitSerDe
> --
>
> Key: HIVE-14867
> URL: https://issues.apache.org/jira/browse/HIVE-14867
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.3.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>
> Create table with MultiDelimitSerde:
> {code}
> CREATE TABLE foo (a string, b string) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' WITH
> SERDEPROPERTIES
> ("field.delim"="|@|","collection.delim"=":","mapkey.delim"="@") stored as
> textfile;
> {code}
> load data into table:
> {code}
> 1|@|Lily|@|HW|@|abc
> 2|@|Lucy|@|LX|@|123
> 3|@|Lilei|@|XX|@|3434
> {code}
> select data from this table:
> {code}
> select * from foo;
> +-++--+
> | foo.a | foo.b |
> +-++--+
> | 1 | Lily^AHW^Aabc|
> | 2 | Lucy^ALX^A123|
> | 3 | Lilei^AXX^A3434 |
> +-++--+
> 3 rows selected (0.905 seconds)
> {code}
> You can see the last column takes all the data, and replace the delimiter to
> default ^A.
> lastColumnTakesRestString should be false by default:
> {code}
> String lastColumnTakesRestString = tbl
> .getProperty(serdeConstants.SERIALIZATION_LAST_COLUMN_TAKES_REST);
> lastColumnTakesRest = (lastColumnTakesRestString != null &&
> lastColumnTakesRestString
> .equalsIgnoreCase("true"));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14582) Add trunc(numeric) udf
[
https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513395#comment-15513395
]
Niklaus Xiao commented on HIVE-14582:
-
Is it possible to add {{trunc(number)}} logic to the existing {{trunc(date)}}
implement ?
> Add trunc(numeric) udf
> --
>
> Key: HIVE-14582
> URL: https://issues.apache.org/jira/browse/HIVE-14582
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Ashutosh Chauhan
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-14582.patch
>
>
> https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14582) Add trunc(numeric) udf
[
https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512983#comment-15512983
]
Niklaus Xiao edited comment on HIVE-14582 at 9/22/16 11:12 AM:
---
Hi [~chinnalalam] bravo job! But I don't think {{trunc2}} is a good name. If
customers need to transform their work from oracle, then they need to modify
all of their script.
was (Author: niklaus.xiao):
Hi [~chinnalalam] bravo job! But I don't think {{trunc2}} is a good name. If
customers need to transform them work from oracle, then they need to modify all
of their script.
> Add trunc(numeric) udf
> --
>
> Key: HIVE-14582
> URL: https://issues.apache.org/jira/browse/HIVE-14582
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Ashutosh Chauhan
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-14582.patch
>
>
> https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14582) Add trunc(numeric) udf
[
https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512983#comment-15512983
]
Niklaus Xiao edited comment on HIVE-14582 at 9/22/16 11:11 AM:
---
Hi [~chinnalalam] bravo job! But I don't think {{trunc2}} is a good name. If
customers need to transform them work from oracle, then they need to modify all
of their script.
was (Author: niklaus.xiao):
Hi [~chinnalalam] bravo job! But I don't think {{trunc2}} is a good name.
> Add trunc(numeric) udf
> --
>
> Key: HIVE-14582
> URL: https://issues.apache.org/jira/browse/HIVE-14582
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Ashutosh Chauhan
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-14582.patch
>
>
> https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14582) Add trunc(numeric) udf
[
https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512983#comment-15512983
]
Niklaus Xiao commented on HIVE-14582:
-
Hi [~chinnalalam] bravo job! But I don't think {{trunc2}} is a good name.
> Add trunc(numeric) udf
> --
>
> Key: HIVE-14582
> URL: https://issues.apache.org/jira/browse/HIVE-14582
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Ashutosh Chauhan
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-14582.patch
>
>
> https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14785) return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
[ https://issues.apache.org/jira/browse/HIVE-14785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499963#comment-15499963 ] Niklaus Xiao commented on HIVE-14785: - return code 2 means Mapreduce task is failed, you need to check the map reduce task log. > return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask > --- > > Key: HIVE-14785 > URL: https://issues.apache.org/jira/browse/HIVE-14785 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.1 > Environment: Hortonworks, Talend , Hive >Reporter: vinitkumar > > Hi, > I am creating partitioned ORS table in Hive using Talend. But after executing > job i am getting error :- > Error while processing statement: FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > Can you please suggest what could be the issue ? > Thanks, > Vinitkumar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14694) UDF rand throws NPE when input data is NULL
[
https://issues.apache.org/jira/browse/HIVE-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15466323#comment-15466323
]
Niklaus Xiao commented on HIVE-14694:
-
[~ashutoshc] ping ?
> UDF rand throws NPE when input data is NULL
> ---
>
> Key: HIVE-14694
> URL: https://issues.apache.org/jira/browse/HIVE-14694
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14694.1.patch
>
>
> When use {{rand}} function with null, HiveServer throws NPE:
> {code}
> 0: jdbc:hive2://10.64.35.144:21066/> desc foo1;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | c1| bigint | |
> +---++--+--+
> 1 row selected (0.075 seconds)
> 0: jdbc:hive2://10.64.35.144:21066/> select * from foo1;
> +--+--+
> | foo1.c1 |
> +--+--+
> | NULL |
> | 1|
> | 2|
> +--+--+
> 3 rows selected (0.124 seconds)
> 0: jdbc:hive2://10.64.35.144:21066/> select rand(c1) from foo1;
> Error: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Unable to execute method public
> org.apache.hadoop.hive.serde2.io.DoubleWritable
> org.apache.hadoop.hive.ql.udf.UDFRand.evaluate(org.apache.hadoop.io.LongWritable)
> on object org.apache.hadoop.hive.ql.udf.UDFRand@37a2b47b of class
> org.apache.hadoop.hive.ql.udf.UDFRand with arguments {null} of size 1
> (state=,code=0)
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1010)
> ... 36 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.udf.UDFRand.evaluate(UDFRand.java:57)
> ... 40 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14694) UDF rand throws NPE when input data is NULL
[
https://issues.apache.org/jira/browse/HIVE-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15462563#comment-15462563
]
Niklaus Xiao commented on HIVE-14694:
-
Unrelated test failures.
> UDF rand throws NPE when input data is NULL
> ---
>
> Key: HIVE-14694
> URL: https://issues.apache.org/jira/browse/HIVE-14694
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14694.1.patch
>
>
> When use {{rand}} function with null, HiveServer throws NPE:
> {code}
> 0: jdbc:hive2://10.64.35.144:21066/> desc foo1;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | c1| bigint | |
> +---++--+--+
> 1 row selected (0.075 seconds)
> 0: jdbc:hive2://10.64.35.144:21066/> select * from foo1;
> +--+--+
> | foo1.c1 |
> +--+--+
> | NULL |
> | 1|
> | 2|
> +--+--+
> 3 rows selected (0.124 seconds)
> 0: jdbc:hive2://10.64.35.144:21066/> select rand(c1) from foo1;
> Error: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Unable to execute method public
> org.apache.hadoop.hive.serde2.io.DoubleWritable
> org.apache.hadoop.hive.ql.udf.UDFRand.evaluate(org.apache.hadoop.io.LongWritable)
> on object org.apache.hadoop.hive.ql.udf.UDFRand@37a2b47b of class
> org.apache.hadoop.hive.ql.udf.UDFRand with arguments {null} of size 1
> (state=,code=0)
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1010)
> ... 36 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.udf.UDFRand.evaluate(UDFRand.java:57)
> ... 40 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14694) UDF rand throws NPE when input data is NULL
[
https://issues.apache.org/jira/browse/HIVE-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-14694:
Attachment: HIVE-14694.1.patch
> UDF rand throws NPE when input data is NULL
> ---
>
> Key: HIVE-14694
> URL: https://issues.apache.org/jira/browse/HIVE-14694
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14694.1.patch
>
>
> When use {{rand}} function with null, HiveServer throws NPE:
> {code}
> 0: jdbc:hive2://10.64.35.144:21066/> desc foo1;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | c1| bigint | |
> +---++--+--+
> 1 row selected (0.075 seconds)
> 0: jdbc:hive2://10.64.35.144:21066/> select * from foo1;
> +--+--+
> | foo1.c1 |
> +--+--+
> | NULL |
> | 1|
> | 2|
> +--+--+
> 3 rows selected (0.124 seconds)
> 0: jdbc:hive2://10.64.35.144:21066/> select rand(c1) from foo1;
> Error: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Unable to execute method public
> org.apache.hadoop.hive.serde2.io.DoubleWritable
> org.apache.hadoop.hive.ql.udf.UDFRand.evaluate(org.apache.hadoop.io.LongWritable)
> on object org.apache.hadoop.hive.ql.udf.UDFRand@37a2b47b of class
> org.apache.hadoop.hive.ql.udf.UDFRand with arguments {null} of size 1
> (state=,code=0)
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1010)
> ... 36 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.udf.UDFRand.evaluate(UDFRand.java:57)
> ... 40 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14694) UDF rand throws NPE when input data is NULL
[
https://issues.apache.org/jira/browse/HIVE-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-14694:
Attachment: (was: HIVE-14694.1.patch)
> UDF rand throws NPE when input data is NULL
> ---
>
> Key: HIVE-14694
> URL: https://issues.apache.org/jira/browse/HIVE-14694
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Fix For: 2.2.0
>
>
> When use {{rand}} function with null, HiveServer throws NPE:
> {code}
> 0: jdbc:hive2://10.64.35.144:21066/> desc foo1;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | c1| bigint | |
> +---++--+--+
> 1 row selected (0.075 seconds)
> 0: jdbc:hive2://10.64.35.144:21066/> select * from foo1;
> +--+--+
> | foo1.c1 |
> +--+--+
> | NULL |
> | 1|
> | 2|
> +--+--+
> 3 rows selected (0.124 seconds)
> 0: jdbc:hive2://10.64.35.144:21066/> select rand(c1) from foo1;
> Error: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Unable to execute method public
> org.apache.hadoop.hive.serde2.io.DoubleWritable
> org.apache.hadoop.hive.ql.udf.UDFRand.evaluate(org.apache.hadoop.io.LongWritable)
> on object org.apache.hadoop.hive.ql.udf.UDFRand@37a2b47b of class
> org.apache.hadoop.hive.ql.udf.UDFRand with arguments {null} of size 1
> (state=,code=0)
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1010)
> ... 36 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.udf.UDFRand.evaluate(UDFRand.java:57)
> ... 40 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14694) UDF rand throws NPE when input data is NULL
[
https://issues.apache.org/jira/browse/HIVE-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-14694:
Fix Version/s: 2.2.0
Status: Patch Available (was: Open)
> UDF rand throws NPE when input data is NULL
> ---
>
> Key: HIVE-14694
> URL: https://issues.apache.org/jira/browse/HIVE-14694
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14694.1.patch
>
>
> When use {{rand}} function with null, HiveServer throws NPE:
> {code}
> 0: jdbc:hive2://10.64.35.144:21066/> desc foo1;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | c1| bigint | |
> +---++--+--+
> 1 row selected (0.075 seconds)
> 0: jdbc:hive2://10.64.35.144:21066/> select * from foo1;
> +--+--+
> | foo1.c1 |
> +--+--+
> | NULL |
> | 1|
> | 2|
> +--+--+
> 3 rows selected (0.124 seconds)
> 0: jdbc:hive2://10.64.35.144:21066/> select rand(c1) from foo1;
> Error: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Unable to execute method public
> org.apache.hadoop.hive.serde2.io.DoubleWritable
> org.apache.hadoop.hive.ql.udf.UDFRand.evaluate(org.apache.hadoop.io.LongWritable)
> on object org.apache.hadoop.hive.ql.udf.UDFRand@37a2b47b of class
> org.apache.hadoop.hive.ql.udf.UDFRand with arguments {null} of size 1
> (state=,code=0)
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1010)
> ... 36 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.udf.UDFRand.evaluate(UDFRand.java:57)
> ... 40 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14694) UDF rand throws NPE when input data is NULL
[
https://issues.apache.org/jira/browse/HIVE-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15457950#comment-15457950
]
Niklaus Xiao commented on HIVE-14694:
-
I tried in MySQL, when input data is null, MySQL will use 0 as the seed
{code}
mysql> select rand(0);
+-+
| rand(0) |
+-+
| 0.15522042769493574 |
+-+
1 row in set (0.00 sec)
mysql> select rand(NULL);
+-+
| rand(NULL) |
+-+
| 0.15522042769493574 |
+-+
1 row in set (0.00 sec)
{code}
So I use 0 as seed when input data is null in the patch, cc [~ashutoshc]
> UDF rand throws NPE when input data is NULL
> ---
>
> Key: HIVE-14694
> URL: https://issues.apache.org/jira/browse/HIVE-14694
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Attachments: HIVE-14694.1.patch
>
>
> When use {{rand}} function with null, HiveServer throws NPE:
> {code}
> 0: jdbc:hive2://10.64.35.144:21066/> desc foo1;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | c1| bigint | |
> +---++--+--+
> 1 row selected (0.075 seconds)
> 0: jdbc:hive2://10.64.35.144:21066/> select * from foo1;
> +--+--+
> | foo1.c1 |
> +--+--+
> | NULL |
> | 1|
> | 2|
> +--+--+
> 3 rows selected (0.124 seconds)
> 0: jdbc:hive2://10.64.35.144:21066/> select rand(c1) from foo1;
> Error: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Unable to execute method public
> org.apache.hadoop.hive.serde2.io.DoubleWritable
> org.apache.hadoop.hive.ql.udf.UDFRand.evaluate(org.apache.hadoop.io.LongWritable)
> on object org.apache.hadoop.hive.ql.udf.UDFRand@37a2b47b of class
> org.apache.hadoop.hive.ql.udf.UDFRand with arguments {null} of size 1
> (state=,code=0)
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1010)
> ... 36 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.udf.UDFRand.evaluate(UDFRand.java:57)
> ... 40 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14694) UDF rand throws NPE when input data is NULL
[
https://issues.apache.org/jira/browse/HIVE-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-14694:
Attachment: HIVE-14694.1.patch
> UDF rand throws NPE when input data is NULL
> ---
>
> Key: HIVE-14694
> URL: https://issues.apache.org/jira/browse/HIVE-14694
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Attachments: HIVE-14694.1.patch
>
>
> When use {{rand}} function with null, HiveServer throws NPE:
> {code}
> 0: jdbc:hive2://10.64.35.144:21066/> desc foo1;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | c1| bigint | |
> +---++--+--+
> 1 row selected (0.075 seconds)
> 0: jdbc:hive2://10.64.35.144:21066/> select * from foo1;
> +--+--+
> | foo1.c1 |
> +--+--+
> | NULL |
> | 1|
> | 2|
> +--+--+
> 3 rows selected (0.124 seconds)
> 0: jdbc:hive2://10.64.35.144:21066/> select rand(c1) from foo1;
> Error: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Unable to execute method public
> org.apache.hadoop.hive.serde2.io.DoubleWritable
> org.apache.hadoop.hive.ql.udf.UDFRand.evaluate(org.apache.hadoop.io.LongWritable)
> on object org.apache.hadoop.hive.ql.udf.UDFRand@37a2b47b of class
> org.apache.hadoop.hive.ql.udf.UDFRand with arguments {null} of size 1
> (state=,code=0)
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1010)
> ... 36 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.udf.UDFRand.evaluate(UDFRand.java:57)
> ... 40 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14658) UDF abs throws NPE when input arg type is string
[
https://issues.apache.org/jira/browse/HIVE-14658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1576#comment-1576
]
Niklaus Xiao commented on HIVE-14658:
-
Unrelated test failures. [~ashutoshc] Could you help to review this ?
> UDF abs throws NPE when input arg type is string
>
>
> Key: HIVE-14658
> URL: https://issues.apache.org/jira/browse/HIVE-14658
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14658.1.patch
>
>
> I know this is not the right use case, but NPE is not exptected.
> {code}
> 0: jdbc:hive2://10.64.35.144:21066/> select abs("foo");
> Error: Error while compiling statement: FAILED: NullPointerException null
> (state=42000,code=4)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14658) UDF abs throws NPE when input arg type is string
[
https://issues.apache.org/jira/browse/HIVE-14658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-14658:
Affects Version/s: 2.2.0
> UDF abs throws NPE when input arg type is string
>
>
> Key: HIVE-14658
> URL: https://issues.apache.org/jira/browse/HIVE-14658
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14658.1.patch
>
>
> I know this is not the right use case, but NPE is not exptected.
> {code}
> 0: jdbc:hive2://10.64.35.144:21066/> select abs("foo");
> Error: Error while compiling statement: FAILED: NullPointerException null
> (state=42000,code=4)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14658) UDF abs throws NPE when input arg type is string
[
https://issues.apache.org/jira/browse/HIVE-14658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-14658:
Fix Version/s: 2.2.0
Status: Patch Available (was: Open)
> UDF abs throws NPE when input arg type is string
>
>
> Key: HIVE-14658
> URL: https://issues.apache.org/jira/browse/HIVE-14658
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.3.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14658.1.patch
>
>
> I know this is not the right use case, but NPE is not exptected.
> {code}
> 0: jdbc:hive2://10.64.35.144:21066/> select abs("foo");
> Error: Error while compiling statement: FAILED: NullPointerException null
> (state=42000,code=4)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14658) UDF abs throws NPE when input arg type is string
[
https://issues.apache.org/jira/browse/HIVE-14658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-14658:
Attachment: HIVE-14658.1.patch
> UDF abs throws NPE when input arg type is string
>
>
> Key: HIVE-14658
> URL: https://issues.apache.org/jira/browse/HIVE-14658
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.3.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14658.1.patch
>
>
> I know this is not the right use case, but NPE is not exptected.
> {code}
> 0: jdbc:hive2://10.64.35.144:21066/> select abs("foo");
> Error: Error while compiling statement: FAILED: NullPointerException null
> (state=42000,code=4)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14570) Create table with column names ROW__ID, INPUT__FILE__NAME, BLOCK__OFFSET__INSIDE__FILE sucess but query fails
[
https://issues.apache.org/jira/browse/HIVE-14570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15427424#comment-15427424
]
Niklaus Xiao commented on HIVE-14570:
-
Unrelated tests failure.
> Create table with column names ROW__ID, INPUT__FILE__NAME,
> BLOCK__OFFSET__INSIDE__FILE sucess but query fails
> -
>
> Key: HIVE-14570
> URL: https://issues.apache.org/jira/browse/HIVE-14570
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Fix For: 2.2.0
>
> Attachments: HIVE-14570.patch
>
>
> {code}
> 0: jdbc:hive2://189.39.151.74:21066/> create table foo1(ROW__ID string);
> No rows affected (0.281 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> create table
> foo2(BLOCK__OFFSET__INSIDE__FILE string);
> No rows affected (0.323 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> create table foo3(INPUT__FILE__NAME
> string);
> No rows affected (0.307 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> select * from foo1;
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'TOK_ALLCOLREF' (state=42000,code=4)
> 0: jdbc:hive2://189.39.151.74:21066/> select * from foo2;
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'TOK_ALLCOLREF' (state=42000,code=4)
> 0: jdbc:hive2://189.39.151.74:21066/> select * from foo3;
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'TOK_ALLCOLREF' (state=42000,code=4)
> {code}
> We should prevent user from creating table with column names the same as
> Virtual Column names
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14570) Create table with column names ROW__ID, INPUT__FILE__NAME, BLOCK__OFFSET__INSIDE__FILE sucess but query fails
[
https://issues.apache.org/jira/browse/HIVE-14570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15426281#comment-15426281
]
Niklaus Xiao commented on HIVE-14570:
-
{code}
0: jdbc:hive2://189.39.151.74:21066/> create table foo3(INPUT__FILE__NAME
string);
Error: Error while compiling statement: FAILED: SemanticException [Error
10316]: Invalid column name INPUT__FILE__NAME (state=42000,code=10316)
{code}
> Create table with column names ROW__ID, INPUT__FILE__NAME,
> BLOCK__OFFSET__INSIDE__FILE sucess but query fails
> -
>
> Key: HIVE-14570
> URL: https://issues.apache.org/jira/browse/HIVE-14570
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Fix For: 2.2.0
>
> Attachments: HIVE-14570.patch
>
>
> {code}
> 0: jdbc:hive2://189.39.151.74:21066/> create table foo1(ROW__ID string);
> No rows affected (0.281 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> create table
> foo2(BLOCK__OFFSET__INSIDE__FILE string);
> No rows affected (0.323 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> create table foo3(INPUT__FILE__NAME
> string);
> No rows affected (0.307 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> select * from foo1;
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'TOK_ALLCOLREF' (state=42000,code=4)
> 0: jdbc:hive2://189.39.151.74:21066/> select * from foo2;
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'TOK_ALLCOLREF' (state=42000,code=4)
> 0: jdbc:hive2://189.39.151.74:21066/> select * from foo3;
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'TOK_ALLCOLREF' (state=42000,code=4)
> {code}
> We should prevent user from creating table with column names the same as
> Virtual Column names
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14570) Create table with column names ROW__ID, INPUT__FILE__NAME, BLOCK__OFFSET__INSIDE__FILE sucess but query fails
[
https://issues.apache.org/jira/browse/HIVE-14570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15426280#comment-15426280
]
Niklaus Xiao commented on HIVE-14570:
-
Attached path check column names when create table.
> Create table with column names ROW__ID, INPUT__FILE__NAME,
> BLOCK__OFFSET__INSIDE__FILE sucess but query fails
> -
>
> Key: HIVE-14570
> URL: https://issues.apache.org/jira/browse/HIVE-14570
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Fix For: 2.2.0
>
> Attachments: HIVE-14570.patch
>
>
> {code}
> 0: jdbc:hive2://189.39.151.74:21066/> create table foo1(ROW__ID string);
> No rows affected (0.281 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> create table
> foo2(BLOCK__OFFSET__INSIDE__FILE string);
> No rows affected (0.323 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> create table foo3(INPUT__FILE__NAME
> string);
> No rows affected (0.307 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> select * from foo1;
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'TOK_ALLCOLREF' (state=42000,code=4)
> 0: jdbc:hive2://189.39.151.74:21066/> select * from foo2;
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'TOK_ALLCOLREF' (state=42000,code=4)
> 0: jdbc:hive2://189.39.151.74:21066/> select * from foo3;
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'TOK_ALLCOLREF' (state=42000,code=4)
> {code}
> We should prevent user from creating table with column names the same as
> Virtual Column names
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14570) Create table with column names ROW__ID, INPUT__FILE__NAME, BLOCK__OFFSET__INSIDE__FILE sucess but query fails
[
https://issues.apache.org/jira/browse/HIVE-14570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-14570:
Fix Version/s: 2.2.0
Status: Patch Available (was: Open)
> Create table with column names ROW__ID, INPUT__FILE__NAME,
> BLOCK__OFFSET__INSIDE__FILE sucess but query fails
> -
>
> Key: HIVE-14570
> URL: https://issues.apache.org/jira/browse/HIVE-14570
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Fix For: 2.2.0
>
> Attachments: HIVE-14570.patch
>
>
> {code}
> 0: jdbc:hive2://189.39.151.74:21066/> create table foo1(ROW__ID string);
> No rows affected (0.281 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> create table
> foo2(BLOCK__OFFSET__INSIDE__FILE string);
> No rows affected (0.323 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> create table foo3(INPUT__FILE__NAME
> string);
> No rows affected (0.307 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> select * from foo1;
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'TOK_ALLCOLREF' (state=42000,code=4)
> 0: jdbc:hive2://189.39.151.74:21066/> select * from foo2;
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'TOK_ALLCOLREF' (state=42000,code=4)
> 0: jdbc:hive2://189.39.151.74:21066/> select * from foo3;
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'TOK_ALLCOLREF' (state=42000,code=4)
> {code}
> We should prevent user from creating table with column names the same as
> Virtual Column names
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14570) Create table with column names ROW__ID, INPUT__FILE__NAME, BLOCK__OFFSET__INSIDE__FILE sucess but query fails
[
https://issues.apache.org/jira/browse/HIVE-14570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-14570:
Attachment: HIVE-14570.patch
> Create table with column names ROW__ID, INPUT__FILE__NAME,
> BLOCK__OFFSET__INSIDE__FILE sucess but query fails
> -
>
> Key: HIVE-14570
> URL: https://issues.apache.org/jira/browse/HIVE-14570
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-14570.patch
>
>
> {code}
> 0: jdbc:hive2://189.39.151.74:21066/> create table foo1(ROW__ID string);
> No rows affected (0.281 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> create table
> foo2(BLOCK__OFFSET__INSIDE__FILE string);
> No rows affected (0.323 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> create table foo3(INPUT__FILE__NAME
> string);
> No rows affected (0.307 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> select * from foo1;
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'TOK_ALLCOLREF' (state=42000,code=4)
> 0: jdbc:hive2://189.39.151.74:21066/> select * from foo2;
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'TOK_ALLCOLREF' (state=42000,code=4)
> 0: jdbc:hive2://189.39.151.74:21066/> select * from foo3;
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'TOK_ALLCOLREF' (state=42000,code=4)
> {code}
> We should prevent user from creating table with column names the same as
> Virtual Column names
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14556) Load data into text table fail caused by IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-14556:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Load data into text table fail caused by IndexOutOfBoundsException
> --
>
> Key: HIVE-14556
> URL: https://issues.apache.org/jira/browse/HIVE-14556
> Project: Hive
> Issue Type: Bug
> Components: File Formats
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Fix For: 2.2.0
>
> Attachments: HIVE-14556.1.patch
>
>
> {code}
> echo "1" > foo.txt
> 0: jdbc:hive2://189.39.151.74:21066/> create table foo(id int) stored as
> textfile;
> No rows affected (1.846 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> load data local inpath '/foo.txt' into
> table foo;
> Error: Error while compiling statement: FAILED: SemanticException Unable to
> load data to destination table. Error: java.lang.IndexOutOfBoundsException
> (state=42000,code=4)
> {code}
> Exception:
> {code}
> 2016-08-17 17:15:36,301 | ERROR | HiveServer2-Handler-Pool: Thread-55 |
> FAILED: SemanticException Unable to load data to destination table. Error:
> java.lang.IndexOutOfBoundsException
> org.apache.hadoop.hive.ql.parse.SemanticException: Unable to load data to
> destination table. Error: java.lang.IndexOutOfBoundsException
> at
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.ensureFileFormatsMatch(LoadSemanticAnalyzer.java:356)
> at
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:236)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:473)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:325)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1358)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1340
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14556) Load data into text table fail caused by IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-14556:
Attachment: HIVE-14556.1.patch
> Load data into text table fail caused by IndexOutOfBoundsException
> --
>
> Key: HIVE-14556
> URL: https://issues.apache.org/jira/browse/HIVE-14556
> Project: Hive
> Issue Type: Bug
> Components: File Formats
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Fix For: 2.2.0
>
> Attachments: HIVE-14556.1.patch
>
>
> {code}
> echo "1" > foo.txt
> 0: jdbc:hive2://189.39.151.74:21066/> create table foo(id int) stored as
> textfile;
> No rows affected (1.846 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> load data local inpath '/foo.txt' into
> table foo;
> Error: Error while compiling statement: FAILED: SemanticException Unable to
> load data to destination table. Error: java.lang.IndexOutOfBoundsException
> (state=42000,code=4)
> {code}
> Exception:
> {code}
> 2016-08-17 17:15:36,301 | ERROR | HiveServer2-Handler-Pool: Thread-55 |
> FAILED: SemanticException Unable to load data to destination table. Error:
> java.lang.IndexOutOfBoundsException
> org.apache.hadoop.hive.ql.parse.SemanticException: Unable to load data to
> destination table. Error: java.lang.IndexOutOfBoundsException
> at
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.ensureFileFormatsMatch(LoadSemanticAnalyzer.java:356)
> at
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:236)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:473)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:325)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1358)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1340
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14556) Load data into text table fail caused by IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-14556:
Fix Version/s: 2.2.0
Status: Patch Available (was: Open)
> Load data into text table fail caused by IndexOutOfBoundsException
> --
>
> Key: HIVE-14556
> URL: https://issues.apache.org/jira/browse/HIVE-14556
> Project: Hive
> Issue Type: Bug
> Components: File Formats
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Fix For: 2.2.0
>
> Attachments: HIVE-14556.1.patch
>
>
> {code}
> echo "1" > foo.txt
> 0: jdbc:hive2://189.39.151.74:21066/> create table foo(id int) stored as
> textfile;
> No rows affected (1.846 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> load data local inpath '/foo.txt' into
> table foo;
> Error: Error while compiling statement: FAILED: SemanticException Unable to
> load data to destination table. Error: java.lang.IndexOutOfBoundsException
> (state=42000,code=4)
> {code}
> Exception:
> {code}
> 2016-08-17 17:15:36,301 | ERROR | HiveServer2-Handler-Pool: Thread-55 |
> FAILED: SemanticException Unable to load data to destination table. Error:
> java.lang.IndexOutOfBoundsException
> org.apache.hadoop.hive.ql.parse.SemanticException: Unable to load data to
> destination table. Error: java.lang.IndexOutOfBoundsException
> at
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.ensureFileFormatsMatch(LoadSemanticAnalyzer.java:356)
> at
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:236)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:473)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:325)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1358)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1340
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14556) Load data into text table fail caused by IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15424194#comment-15424194
]
Niklaus Xiao commented on HIVE-14556:
-
When load a file which is small than 3 bytes, this issue happens.
> Load data into text table fail caused by IndexOutOfBoundsException
> --
>
> Key: HIVE-14556
> URL: https://issues.apache.org/jira/browse/HIVE-14556
> Project: Hive
> Issue Type: Bug
> Components: File Formats
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>
> {code}
> echo "1" > foo.txt
> 0: jdbc:hive2://189.39.151.74:21066/> create table foo(id int) stored as
> textfile;
> No rows affected (1.846 seconds)
> 0: jdbc:hive2://189.39.151.74:21066/> load data local inpath '/foo.txt' into
> table foo;
> Error: Error while compiling statement: FAILED: SemanticException Unable to
> load data to destination table. Error: java.lang.IndexOutOfBoundsException
> (state=42000,code=4)
> {code}
> Exception:
> {code}
> 2016-08-17 17:15:36,301 | ERROR | HiveServer2-Handler-Pool: Thread-55 |
> FAILED: SemanticException Unable to load data to destination table. Error:
> java.lang.IndexOutOfBoundsException
> org.apache.hadoop.hive.ql.parse.SemanticException: Unable to load data to
> destination table. Error: java.lang.IndexOutOfBoundsException
> at
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.ensureFileFormatsMatch(LoadSemanticAnalyzer.java:356)
> at
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:236)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:473)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:325)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1358)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1340
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14411) selecting Hive on Hbase table may cause FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408758#comment-15408758
]
Niklaus Xiao commented on HIVE-14411:
-
Test failures not related.
> selecting Hive on Hbase table may cause FileNotFoundException
> -
>
> Key: HIVE-14411
> URL: https://issues.apache.org/jira/browse/HIVE-14411
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.3.0
>Reporter: Rudd Chen
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-14411.1.patch
>
>
> 1. create a Hbase table hbase_table
> 2. create a external Hive table test_table mapping to the hbase table
> example:
> create 'hbase_t'
> ,{NAME=>'cf',COMPRESSION=>'snappy'},{NUMREGIONS=>15,SPLITALGO=>'HexStringSplit'}
> create external table hbase_t_hive(key1 string,cf_train string,cf_flight
> string,cf_wbsw string,cf_wbxw string,cf_bgrz string,cf_bgtf string)
> stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> with
> serdeproperties("hbase.columns.mapping"=":key,cf:train,cf:flight,cf:wbsw,cf:wbxw,cf:bgrz,cf:bgtf")
> tblproperties("hbase.table.name"="hbase_t");
> create table test3 as select * from hbase_t_hive where 1=2;
>
> if hive.optimize.null.scan=true, it will return an FileNotFoundException
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-14304) Beeline command will fail when entireLineAsCommand set to true
[
https://issues.apache.org/jira/browse/HIVE-14304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao reassigned HIVE-14304:
---
Assignee: Hari Sankar Sivarama Subramaniyan (was: Niklaus Xiao)
> Beeline command will fail when entireLineAsCommand set to true
> --
>
> Key: HIVE-14304
> URL: https://issues.apache.org/jira/browse/HIVE-14304
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Hari Sankar Sivarama Subramaniyan
> Fix For: 2.2.0
>
> Attachments: HIVE-14304.1.patch
>
>
> Use beeline
> {code}
> beeline --entireLineAsCommand=true
> {code}
> show tables fail:
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> show tables;
> Error: Error while compiling statement: FAILED: ParseException line 1:11
> extraneous input ';' expecting EOF near '' (state=42000,code=4)
> {code}
> We should remove the trailing semi-colon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-14411) selecting Hive on Hbase table may cause FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao reassigned HIVE-14411:
---
Assignee: Ashutosh Chauhan (was: Niklaus Xiao)
> selecting Hive on Hbase table may cause FileNotFoundException
> -
>
> Key: HIVE-14411
> URL: https://issues.apache.org/jira/browse/HIVE-14411
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.3.0
>Reporter: Rudd Chen
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-14411.1.patch
>
>
> 1. create a Hbase table hbase_table
> 2. create a external Hive table test_table mapping to the hbase table
> example:
> create 'hbase_t'
> ,{NAME=>'cf',COMPRESSION=>'snappy'},{NUMREGIONS=>15,SPLITALGO=>'HexStringSplit'}
> create external table hbase_t_hive(key1 string,cf_train string,cf_flight
> string,cf_wbsw string,cf_wbxw string,cf_bgrz string,cf_bgtf string)
> stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> with
> serdeproperties("hbase.columns.mapping"=":key,cf:train,cf:flight,cf:wbsw,cf:wbxw,cf:bgrz,cf:bgtf")
> tblproperties("hbase.table.name"="hbase_t");
> create table test3 as select * from hbase_t_hive where 1=2;
>
> if hive.optimize.null.scan=true, it will return an FileNotFoundException
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14411) selecting Hive on Hbase table may cause FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407591#comment-15407591
]
Niklaus Xiao commented on HIVE-14411:
-
We should not apply NullScanOptimizer for non-native table.
cc [~ashutoshc] for code review.
> selecting Hive on Hbase table may cause FileNotFoundException
> -
>
> Key: HIVE-14411
> URL: https://issues.apache.org/jira/browse/HIVE-14411
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.3.0
>Reporter: Rudd Chen
>Assignee: Niklaus Xiao
> Attachments: HIVE-14411.1.patch
>
>
> 1. create a Hbase table hbase_table
> 2. create a external Hive table test_table mapping to the hbase table
> example:
> create 'hbase_t'
> ,{NAME=>'cf',COMPRESSION=>'snappy'},{NUMREGIONS=>15,SPLITALGO=>'HexStringSplit'}
> create external table hbase_t_hive(key1 string,cf_train string,cf_flight
> string,cf_wbsw string,cf_wbxw string,cf_bgrz string,cf_bgtf string)
> stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> with
> serdeproperties("hbase.columns.mapping"=":key,cf:train,cf:flight,cf:wbsw,cf:wbxw,cf:bgrz,cf:bgtf")
> tblproperties("hbase.table.name"="hbase_t");
> create table test3 as select * from hbase_t_hive where 1=2;
>
> if hive.optimize.null.scan=true, it will return an FileNotFoundException
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14411) selecting Hive on Hbase table may cause FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-14411:
Attachment: HIVE-14411.1.patch
> selecting Hive on Hbase table may cause FileNotFoundException
> -
>
> Key: HIVE-14411
> URL: https://issues.apache.org/jira/browse/HIVE-14411
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.3.0
>Reporter: Rudd Chen
>Assignee: Niklaus Xiao
> Attachments: HIVE-14411.1.patch
>
>
> 1. create a Hbase table hbase_table
> 2. create a external Hive table test_table mapping to the hbase table
> example:
> create 'hbase_t'
> ,{NAME=>'cf',COMPRESSION=>'snappy'},{NUMREGIONS=>15,SPLITALGO=>'HexStringSplit'}
> create external table hbase_t_hive(key1 string,cf_train string,cf_flight
> string,cf_wbsw string,cf_wbxw string,cf_bgrz string,cf_bgtf string)
> stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> with
> serdeproperties("hbase.columns.mapping"=":key,cf:train,cf:flight,cf:wbsw,cf:wbxw,cf:bgrz,cf:bgtf")
> tblproperties("hbase.table.name"="hbase_t");
> create table test3 as select * from hbase_t_hive where 1=2;
>
> if hive.optimize.null.scan=true, it will return an FileNotFoundException
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14411) selecting Hive on Hbase table may cause FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao updated HIVE-14411:
Target Version/s: 2.2.0
Status: Patch Available (was: Open)
> selecting Hive on Hbase table may cause FileNotFoundException
> -
>
> Key: HIVE-14411
> URL: https://issues.apache.org/jira/browse/HIVE-14411
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.3.0
>Reporter: Rudd Chen
>Assignee: Niklaus Xiao
> Attachments: HIVE-14411.1.patch
>
>
> 1. create a Hbase table hbase_table
> 2. create a external Hive table test_table mapping to the hbase table
> example:
> create 'hbase_t'
> ,{NAME=>'cf',COMPRESSION=>'snappy'},{NUMREGIONS=>15,SPLITALGO=>'HexStringSplit'}
> create external table hbase_t_hive(key1 string,cf_train string,cf_flight
> string,cf_wbsw string,cf_wbxw string,cf_bgrz string,cf_bgtf string)
> stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> with
> serdeproperties("hbase.columns.mapping"=":key,cf:train,cf:flight,cf:wbsw,cf:wbxw,cf:bgrz,cf:bgtf")
> tblproperties("hbase.table.name"="hbase_t");
> create table test3 as select * from hbase_t_hive where 1=2;
>
> if hive.optimize.null.scan=true, it will return an FileNotFoundException
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-14411) selecting Hive on Hbase table may cause FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niklaus Xiao reassigned HIVE-14411:
---
Assignee: Niklaus Xiao
> selecting Hive on Hbase table may cause FileNotFoundException
> -
>
> Key: HIVE-14411
> URL: https://issues.apache.org/jira/browse/HIVE-14411
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.3.0
>Reporter: Rudd Chen
>Assignee: Niklaus Xiao
>
> 1. create a Hbase table hbase_table
> 2. create a external Hive table test_table mapping to the hbase table
> example:
> create 'hbase_t'
> ,{NAME=>'cf',COMPRESSION=>'snappy'},{NUMREGIONS=>15,SPLITALGO=>'HexStringSplit'}
> create external table hbase_t_hive(key1 string,cf_train string,cf_flight
> string,cf_wbsw string,cf_wbxw string,cf_bgrz string,cf_bgtf string)
> stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> with
> serdeproperties("hbase.columns.mapping"=":key,cf:train,cf:flight,cf:wbsw,cf:wbxw,cf:bgrz,cf:bgtf")
> tblproperties("hbase.table.name"="hbase_t");
> create table test3 as select * from hbase_t_hive where 1=2;
>
> if hive.optimize.null.scan=true, it will return an FileNotFoundException
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14304) Beeline command will fail when entireLineAsCommand set to true
[
https://issues.apache.org/jira/browse/HIVE-14304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389115#comment-15389115
]
niklaus xiao commented on HIVE-14304:
-
cc [~hsubramaniyan] for code review, since you are the original author.
> Beeline command will fail when entireLineAsCommand set to true
> --
>
> Key: HIVE-14304
> URL: https://issues.apache.org/jira/browse/HIVE-14304
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.3.0, 2.2.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
> Fix For: 2.2.0
>
> Attachments: HIVE-14304.1.patch
>
>
> Use beeline
> {code}
> beeline --entireLineAsCommand=true
> {code}
> show tables fail:
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> show tables;
> Error: Error while compiling statement: FAILED: ParseException line 1:11
> extraneous input ';' expecting EOF near '' (state=42000,code=4)
> {code}
> We should remove the trailing semi-colon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14304) Beeline command will fail when entireLineAsCommand set to true
[
https://issues.apache.org/jira/browse/HIVE-14304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14304:
Fix Version/s: 2.2.0
Status: Patch Available (was: Open)
> Beeline command will fail when entireLineAsCommand set to true
> --
>
> Key: HIVE-14304
> URL: https://issues.apache.org/jira/browse/HIVE-14304
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.3.0, 2.2.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
> Fix For: 2.2.0
>
> Attachments: HIVE-14304.1.patch
>
>
> Use beeline
> {code}
> beeline --entireLineAsCommand=true
> {code}
> show tables fail:
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> show tables;
> Error: Error while compiling statement: FAILED: ParseException line 1:11
> extraneous input ';' expecting EOF near '' (state=42000,code=4)
> {code}
> We should remove the trailing semi-colon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14304) Beeline command will fail when entireLineAsCommand set to true
[
https://issues.apache.org/jira/browse/HIVE-14304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14304:
Attachment: HIVE-14304.1.patch
> Beeline command will fail when entireLineAsCommand set to true
> --
>
> Key: HIVE-14304
> URL: https://issues.apache.org/jira/browse/HIVE-14304
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.3.0, 2.2.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
> Attachments: HIVE-14304.1.patch
>
>
> Use beeline
> {code}
> beeline --entireLineAsCommand=true
> {code}
> show tables fail:
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> show tables;
> Error: Error while compiling statement: FAILED: ParseException line 1:11
> extraneous input ';' expecting EOF near '' (state=42000,code=4)
> {code}
> We should remove the trailing semi-colon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14295) Some metastore event listeners always initialize deleteData as false
[
https://issues.apache.org/jira/browse/HIVE-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386908#comment-15386908
]
niklaus xiao commented on HIVE-14295:
-
Should be 2.2
> Some metastore event listeners always initialize deleteData as false
>
>
> Key: HIVE-14295
> URL: https://issues.apache.org/jira/browse/HIVE-14295
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.1.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14295.1.patch
>
>
> DropTableEvent:
> {code}
> public DropTableEvent(Table table, boolean status, boolean deleteData,
> HMSHandler handler) {
> super(status, handler);
> this.table = table;
> // In HiveMetaStore, the deleteData flag indicates whether DFS data
> should be
> // removed on a drop.
> this.deleteData = false;
> }
> {code}
> Same as PreDropPartitionEvent and PreDropTableEvent
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14295) Some metastore event listeners always initialize deleteData as false
[
https://issues.apache.org/jira/browse/HIVE-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14295:
Fix Version/s: (was: 1.3.0)
2.2.0
> Some metastore event listeners always initialize deleteData as false
>
>
> Key: HIVE-14295
> URL: https://issues.apache.org/jira/browse/HIVE-14295
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.1.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14295.1.patch
>
>
> DropTableEvent:
> {code}
> public DropTableEvent(Table table, boolean status, boolean deleteData,
> HMSHandler handler) {
> super(status, handler);
> this.table = table;
> // In HiveMetaStore, the deleteData flag indicates whether DFS data
> should be
> // removed on a drop.
> this.deleteData = false;
> }
> {code}
> Same as PreDropPartitionEvent and PreDropTableEvent
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14295) Some metastore event listeners always initialize deleteData as false
[
https://issues.apache.org/jira/browse/HIVE-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385717#comment-15385717
]
niklaus xiao commented on HIVE-14295:
-
Small patch. Could you take a look ? [~ashutoshc] Thank you.
> Some metastore event listeners always initialize deleteData as false
>
>
> Key: HIVE-14295
> URL: https://issues.apache.org/jira/browse/HIVE-14295
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.1.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
>Priority: Minor
> Fix For: 1.3.0
>
> Attachments: HIVE-14295.1.patch
>
>
> DropTableEvent:
> {code}
> public DropTableEvent(Table table, boolean status, boolean deleteData,
> HMSHandler handler) {
> super(status, handler);
> this.table = table;
> // In HiveMetaStore, the deleteData flag indicates whether DFS data
> should be
> // removed on a drop.
> this.deleteData = false;
> }
> {code}
> Same as PreDropPartitionEvent and PreDropTableEvent
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14295) Some metastore event listeners always initialize deleteData as false
[
https://issues.apache.org/jira/browse/HIVE-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14295:
Attachment: HIVE-14295.1.patch
> Some metastore event listeners always initialize deleteData as false
>
>
> Key: HIVE-14295
> URL: https://issues.apache.org/jira/browse/HIVE-14295
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.1.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
>Priority: Minor
> Fix For: 1.3.0
>
> Attachments: HIVE-14295.1.patch
>
>
> DropTableEvent:
> {code}
> public DropTableEvent(Table table, boolean status, boolean deleteData,
> HMSHandler handler) {
> super(status, handler);
> this.table = table;
> // In HiveMetaStore, the deleteData flag indicates whether DFS data
> should be
> // removed on a drop.
> this.deleteData = false;
> }
> {code}
> Same as PreDropPartitionEvent and PreDropTableEvent
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14295) Some metastore event listeners always initialize deleteData as false
[
https://issues.apache.org/jira/browse/HIVE-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14295:
Fix Version/s: 1.3.0
Status: Patch Available (was: Open)
> Some metastore event listeners always initialize deleteData as false
>
>
> Key: HIVE-14295
> URL: https://issues.apache.org/jira/browse/HIVE-14295
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0, 1.3.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
>Priority: Minor
> Fix For: 1.3.0
>
> Attachments: HIVE-14295.1.patch
>
>
> DropTableEvent:
> {code}
> public DropTableEvent(Table table, boolean status, boolean deleteData,
> HMSHandler handler) {
> super(status, handler);
> this.table = table;
> // In HiveMetaStore, the deleteData flag indicates whether DFS data
> should be
> // removed on a drop.
> this.deleteData = false;
> }
> {code}
> Same as PreDropPartitionEvent and PreDropTableEvent
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13883) WebHCat leaves token crc file never gets deleted
[
https://issues.apache.org/jira/browse/HIVE-13883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382093#comment-15382093
]
niklaus xiao commented on HIVE-13883:
-
Test failure not related.
> WebHCat leaves token crc file never gets deleted
>
>
> Key: HIVE-13883
> URL: https://issues.apache.org/jira/browse/HIVE-13883
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Affects Versions: 1.2.0, 1.1.1, 1.2.1, 2.0.1
>Reporter: niklaus xiao
>Priority: Minor
> Attachments: HIVE-13883.2.patch, HIVE-13883.3.patch, HIVE-13883.patch
>
>
> In one of our long run environment, there are thousands of
> /tmp/.templeton*.tmp.crc files,
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc
> ...
> -rw-r--r-- 1 omm wheel 12 May 26 18:15
> /tmp/.templeton6676048390600607654.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:14
> /tmp/.templeton2733383617337556503.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:12
> /tmp/.templeton2183121761801669064.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:11
> /tmp/.templeton2689764046140543879.tmp.crc
> ...
> {quote}
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc | wc -l
> 17986
> {quote}
> It's created by webhcat,
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L193]
> and never gets deleted
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L110]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-14248) query with view in union adds underlying table as direct input
[
https://issues.apache.org/jira/browse/HIVE-14248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao resolved HIVE-14248.
-
Resolution: Duplicate
> query with view in union adds underlying table as direct input
> --
>
> Key: HIVE-14248
> URL: https://issues.apache.org/jira/browse/HIVE-14248
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.3.0
>Reporter: niklaus xiao
>
> In the following case,
> {code}
> create view V as select * from T;
> select * from V union all select * from V
> {code}
> The semantic analyzer inputs contain input table T as a direct input instead
> of adding it as an indirect input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14248) query with view in union adds underlying table as direct input
[
https://issues.apache.org/jira/browse/HIVE-14248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382085#comment-15382085
]
niklaus xiao commented on HIVE-14248:
-
Fixed by [HIVE-13991] .
How can i close this issue ?
> query with view in union adds underlying table as direct input
> --
>
> Key: HIVE-14248
> URL: https://issues.apache.org/jira/browse/HIVE-14248
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.3.0
>Reporter: niklaus xiao
>
> In the following case,
> {code}
> create view V as select * from T;
> select * from V union all select * from V
> {code}
> The semantic analyzer inputs contain input table T as a direct input instead
> of adding it as an indirect input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-14248) query with view in union adds underlying table as direct input
[
https://issues.apache.org/jira/browse/HIVE-14248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14248:
Comment: was deleted
(was: The same issue like
[HIVE-10875|https://issues.apache.org/jira/browse/HIVE-10875].
Could you help to take a look? [~thejas])
> query with view in union adds underlying table as direct input
> --
>
> Key: HIVE-14248
> URL: https://issues.apache.org/jira/browse/HIVE-14248
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.3.0
>Reporter: niklaus xiao
>
> In the following case,
> {code}
> create view V as select * from T;
> select * from V union all select * from V
> {code}
> The semantic analyzer inputs contain input table T as a direct input instead
> of adding it as an indirect input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13883) WebHCat leaves token crc file never gets deleted
[
https://issues.apache.org/jira/browse/HIVE-13883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-13883:
Attachment: HIVE-13883.3.patch
> WebHCat leaves token crc file never gets deleted
>
>
> Key: HIVE-13883
> URL: https://issues.apache.org/jira/browse/HIVE-13883
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Affects Versions: 1.2.0, 1.1.1, 1.2.1, 2.0.1
>Reporter: niklaus xiao
>Priority: Minor
> Attachments: HIVE-13883.2.patch, HIVE-13883.3.patch, HIVE-13883.patch
>
>
> In one of our long run environment, there are thousands of
> /tmp/.templeton*.tmp.crc files,
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc
> ...
> -rw-r--r-- 1 omm wheel 12 May 26 18:15
> /tmp/.templeton6676048390600607654.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:14
> /tmp/.templeton2733383617337556503.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:12
> /tmp/.templeton2183121761801669064.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:11
> /tmp/.templeton2689764046140543879.tmp.crc
> ...
> {quote}
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc | wc -l
> 17986
> {quote}
> It's created by webhcat,
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L193]
> and never gets deleted
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L110]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13883) WebHCat leaves token crc file never gets deleted
[
https://issues.apache.org/jira/browse/HIVE-13883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15381889#comment-15381889
]
niklaus xiao commented on HIVE-13883:
-
Rebased.
> WebHCat leaves token crc file never gets deleted
>
>
> Key: HIVE-13883
> URL: https://issues.apache.org/jira/browse/HIVE-13883
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Affects Versions: 1.2.0, 1.1.1, 1.2.1, 2.0.1
>Reporter: niklaus xiao
>Priority: Minor
> Attachments: HIVE-13883.2.patch, HIVE-13883.3.patch, HIVE-13883.patch
>
>
> In one of our long run environment, there are thousands of
> /tmp/.templeton*.tmp.crc files,
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc
> ...
> -rw-r--r-- 1 omm wheel 12 May 26 18:15
> /tmp/.templeton6676048390600607654.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:14
> /tmp/.templeton2733383617337556503.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:12
> /tmp/.templeton2183121761801669064.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:11
> /tmp/.templeton2689764046140543879.tmp.crc
> ...
> {quote}
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc | wc -l
> 17986
> {quote}
> It's created by webhcat,
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L193]
> and never gets deleted
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L110]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13883) WebHCat leaves token crc file never gets deleted
[
https://issues.apache.org/jira/browse/HIVE-13883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379053#comment-15379053
]
niklaus xiao commented on HIVE-13883:
-
Thank you for your review, updated another patch.
> WebHCat leaves token crc file never gets deleted
>
>
> Key: HIVE-13883
> URL: https://issues.apache.org/jira/browse/HIVE-13883
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Affects Versions: 1.2.0, 1.1.1, 1.2.1, 2.0.1
>Reporter: niklaus xiao
>Priority: Minor
> Attachments: HIVE-13883.2.patch, HIVE-13883.patch
>
>
> In one of our long run environment, there are thousands of
> /tmp/.templeton*.tmp.crc files,
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc
> ...
> -rw-r--r-- 1 omm wheel 12 May 26 18:15
> /tmp/.templeton6676048390600607654.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:14
> /tmp/.templeton2733383617337556503.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:12
> /tmp/.templeton2183121761801669064.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:11
> /tmp/.templeton2689764046140543879.tmp.crc
> ...
> {quote}
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc | wc -l
> 17986
> {quote}
> It's created by webhcat,
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L193]
> and never gets deleted
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L110]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13883) WebHCat leaves token crc file never gets deleted
[
https://issues.apache.org/jira/browse/HIVE-13883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-13883:
Attachment: HIVE-13883.2.patch
> WebHCat leaves token crc file never gets deleted
>
>
> Key: HIVE-13883
> URL: https://issues.apache.org/jira/browse/HIVE-13883
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Affects Versions: 1.2.0, 1.1.1, 1.2.1, 2.0.1
>Reporter: niklaus xiao
>Priority: Minor
> Attachments: HIVE-13883.2.patch, HIVE-13883.patch
>
>
> In one of our long run environment, there are thousands of
> /tmp/.templeton*.tmp.crc files,
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc
> ...
> -rw-r--r-- 1 omm wheel 12 May 26 18:15
> /tmp/.templeton6676048390600607654.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:14
> /tmp/.templeton2733383617337556503.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:12
> /tmp/.templeton2183121761801669064.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:11
> /tmp/.templeton2689764046140543879.tmp.crc
> ...
> {quote}
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc | wc -l
> 17986
> {quote}
> It's created by webhcat,
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L193]
> and never gets deleted
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L110]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14248) query with view in union adds underlying table as direct input
[
https://issues.apache.org/jira/browse/HIVE-14248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14248:
Component/s: Authorization
> query with view in union adds underlying table as direct input
> --
>
> Key: HIVE-14248
> URL: https://issues.apache.org/jira/browse/HIVE-14248
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.3.0
>Reporter: niklaus xiao
>
> In the following case,
> {code}
> create view V as select * from T;
> select * from V union all select * from V
> {code}
> The semantic analyzer inputs contain input table T as a direct input instead
> of adding it as an indirect input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14248) query with view in union adds underlying table as direct input
[
https://issues.apache.org/jira/browse/HIVE-14248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14248:
Affects Version/s: 1.3.0
> query with view in union adds underlying table as direct input
> --
>
> Key: HIVE-14248
> URL: https://issues.apache.org/jira/browse/HIVE-14248
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: niklaus xiao
>
> In the following case,
> {code}
> create view V as select * from T;
> select * from V union all select * from V
> {code}
> The semantic analyzer inputs contain input table T as a direct input instead
> of adding it as an indirect input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14248) query with view in union adds underlying table as direct input
[
https://issues.apache.org/jira/browse/HIVE-14248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379030#comment-15379030
]
niklaus xiao edited comment on HIVE-14248 at 7/15/16 8:49 AM:
--
The same issue like
[HIVE-10875|https://issues.apache.org/jira/browse/HIVE-10875].
Could you help to take a look? [~thejas]
was (Author: niklaus.xiao):
The same issue as [HIVE-10875|https://issues.apache.org/jira/browse/HIVE-10875].
Could you help to take a look? [~thejas]
> query with view in union adds underlying table as direct input
> --
>
> Key: HIVE-14248
> URL: https://issues.apache.org/jira/browse/HIVE-14248
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: niklaus xiao
>
> In the following case,
> {code}
> create view V as select * from T;
> select * from V union all select * from V
> {code}
> The semantic analyzer inputs contain input table T as a direct input instead
> of adding it as an indirect input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14248) query with view in union adds underlying table as direct input
[
https://issues.apache.org/jira/browse/HIVE-14248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379030#comment-15379030
]
niklaus xiao commented on HIVE-14248:
-
The same issue as [HIVE-10875|https://issues.apache.org/jira/browse/HIVE-10875].
Could you help to take a look? [~thejas]
> query with view in union adds underlying table as direct input
> --
>
> Key: HIVE-14248
> URL: https://issues.apache.org/jira/browse/HIVE-14248
> Project: Hive
> Issue Type: Bug
>Reporter: niklaus xiao
>
> In the following case,
> {code}
> create view V as select * from T;
> select * from V union all select * from V
> {code}
> The semantic analyzer inputs contain input table T as a direct input instead
> of adding it as an indirect input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10022) Authorization checks for non existent file/directory should not be recursive
[
https://issues.apache.org/jira/browse/HIVE-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378922#comment-15378922
]
niklaus xiao commented on HIVE-10022:
-
Regex works with LOAD DATA.
{code}
0: jdbc:hive2://189.39.151.74:21066/> dfs -ls /tmp/data;
+---+--+
|DFS Output |
+---+--+
| Found 3 items |
| -rw--- 3 admin hadoop 0 2016-07-15 14:48 /tmp/data/1.txt |
| -rw--- 3 admin hadoop 0 2016-07-15 14:48 /tmp/data/2.txt |
| -rw--- 3 admin hadoop 0 2016-07-15 14:47 /tmp/data/3.info |
+---+--+
4 rows selected (0.029 seconds)
0: jdbc:hive2://189.39.151.74:21066/> load data inpath '/tmp/data/*.txt' into
table foo;
INFO : Loading data to table db2.foo from hdfs://hacluster/tmp/data/*.txt
No rows affected (0.558 seconds)
0: jdbc:hive2://189.39.151.74:21066/> dfs -ls /user/hive/warehouse/db2.db/foo;
+--+--+
| DFS Output
|
+--+--+
| Found 2 items
|
| -rwxrwx---+ 3 admin hive 0 2016-07-15 14:48
/user/hive/warehouse/db2.db/foo/1.txt |
| -rwxrwx---+ 3 admin hive 0 2016-07-15 14:48
/user/hive/warehouse/db2.db/foo/2.txt |
+--+--+
3 rows selected (0.091 seconds)
0: jdbc:hive2://189.39.151.74:21066/> dfs -ls /tmp/data;
+---+--+
|DFS Output |
+---+--+
| Found 1 items |
| -rw--- 3 admin hadoop 0 2016-07-15 14:47 /tmp/data/3.info |
+---+--+
2 rows selected (0.047 seconds)
0: jdbc:hive2://189.39.151.74:21066/>
{code}
Expand the regex and check on the regex match might be a good solution.
> Authorization checks for non existent file/directory should not be recursive
>
>
> Key: HIVE-10022
> URL: https://issues.apache.org/jira/browse/HIVE-10022
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 0.14.0
>Reporter: Pankit Thapar
>Assignee: Pankit Thapar
> Attachments: HIVE-10022.2.patch, HIVE-10022.3.patch, HIVE-10022.patch
>
>
> I am testing a query like :
> set hive.test.authz.sstd.hs2.mode=true;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set
> hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateConfigUserAuthenticator;
> set hive.security.authorization.enabled=true;
> set user.name=user1;
> create table auth_noupd(i int) clustered by (i) into 2 buckets stored as orc
> location '${OUTPUT}' TBLPROPERTIES ('transactional'='true');
> Now, in the above query, since authorization is true,
> we would end up calling doAuthorizationV2() which ultimately ends up calling
> SQLAuthorizationUtils.getPrivilegesFromFS() which calls a recursive method :
> FileUtils.isActionPermittedForFileHierarchy() with the object or the ancestor
> of the object we are trying to authorize if the object does not exist.
> The logic in FileUtils.isActionPermittedForFileHierarchy() is DFS.
> Now assume, we have a path as a/b/c/d that we are trying to authorize.
> In case, a/b/c/d does not exist, we would call
> FileUtils.isActionPermittedForFileHierarchy() with say a/b/ assuming a/b/c
> also does not exist.
> If under the subtree at a/b, we have millions of files, then
> FileUtils.isActionPermittedForFileHierarchy() is going to check file
> permission on each of those objects.
> I do not completely understand why do we have to check for file permissions
> in all the objects in branch of the tree that we are not trying to read
> from /write to.
> We could have checked file permission on the ancestor that exists and if it
> matches what we expect, the return true.
> Please confirm if this is a bug so that I can submit a patch else l
[jira] [Comment Edited] (HIVE-10022) Authorization checks for non existent file/directory should not be recursive
[
https://issues.apache.org/jira/browse/HIVE-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378820#comment-15378820
]
niklaus xiao edited comment on HIVE-10022 at 7/15/16 4:10 AM:
--
What if a query like load data inpath which the path is a regex:
{code}
LOAD DATA INPATH '/tmp/data/*.txt' INTO TABLE foo;
{code}
'/tmp/data/*.txt' does not exists, but maybe we should check /tmp/data/
recursive.
was (Author: niklaus.xiao):
What if a query like load data inpath which the path is a regex:
{code}
LOAD DATA INPATH '/tmp/data/*.txt' INTO TABLE foo;
{code}
> Authorization checks for non existent file/directory should not be recursive
>
>
> Key: HIVE-10022
> URL: https://issues.apache.org/jira/browse/HIVE-10022
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 0.14.0
>Reporter: Pankit Thapar
>Assignee: Pankit Thapar
> Attachments: HIVE-10022.2.patch, HIVE-10022.3.patch, HIVE-10022.patch
>
>
> I am testing a query like :
> set hive.test.authz.sstd.hs2.mode=true;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set
> hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateConfigUserAuthenticator;
> set hive.security.authorization.enabled=true;
> set user.name=user1;
> create table auth_noupd(i int) clustered by (i) into 2 buckets stored as orc
> location '${OUTPUT}' TBLPROPERTIES ('transactional'='true');
> Now, in the above query, since authorization is true,
> we would end up calling doAuthorizationV2() which ultimately ends up calling
> SQLAuthorizationUtils.getPrivilegesFromFS() which calls a recursive method :
> FileUtils.isActionPermittedForFileHierarchy() with the object or the ancestor
> of the object we are trying to authorize if the object does not exist.
> The logic in FileUtils.isActionPermittedForFileHierarchy() is DFS.
> Now assume, we have a path as a/b/c/d that we are trying to authorize.
> In case, a/b/c/d does not exist, we would call
> FileUtils.isActionPermittedForFileHierarchy() with say a/b/ assuming a/b/c
> also does not exist.
> If under the subtree at a/b, we have millions of files, then
> FileUtils.isActionPermittedForFileHierarchy() is going to check file
> permission on each of those objects.
> I do not completely understand why do we have to check for file permissions
> in all the objects in branch of the tree that we are not trying to read
> from /write to.
> We could have checked file permission on the ancestor that exists and if it
> matches what we expect, the return true.
> Please confirm if this is a bug so that I can submit a patch else let me know
> what I am missing ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10022) Authorization checks for non existent file/directory should not be recursive
[
https://issues.apache.org/jira/browse/HIVE-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378820#comment-15378820
]
niklaus xiao commented on HIVE-10022:
-
What if a query like load data inpath which the path is a regex:
{code}
LOAD DATA INPATH '/tmp/data/*.txt' INTO TABLE foo;
{code}
> Authorization checks for non existent file/directory should not be recursive
>
>
> Key: HIVE-10022
> URL: https://issues.apache.org/jira/browse/HIVE-10022
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 0.14.0
>Reporter: Pankit Thapar
>Assignee: Pankit Thapar
> Attachments: HIVE-10022.2.patch, HIVE-10022.3.patch, HIVE-10022.patch
>
>
> I am testing a query like :
> set hive.test.authz.sstd.hs2.mode=true;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set
> hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateConfigUserAuthenticator;
> set hive.security.authorization.enabled=true;
> set user.name=user1;
> create table auth_noupd(i int) clustered by (i) into 2 buckets stored as orc
> location '${OUTPUT}' TBLPROPERTIES ('transactional'='true');
> Now, in the above query, since authorization is true,
> we would end up calling doAuthorizationV2() which ultimately ends up calling
> SQLAuthorizationUtils.getPrivilegesFromFS() which calls a recursive method :
> FileUtils.isActionPermittedForFileHierarchy() with the object or the ancestor
> of the object we are trying to authorize if the object does not exist.
> The logic in FileUtils.isActionPermittedForFileHierarchy() is DFS.
> Now assume, we have a path as a/b/c/d that we are trying to authorize.
> In case, a/b/c/d does not exist, we would call
> FileUtils.isActionPermittedForFileHierarchy() with say a/b/ assuming a/b/c
> also does not exist.
> If under the subtree at a/b, we have millions of files, then
> FileUtils.isActionPermittedForFileHierarchy() is going to check file
> permission on each of those objects.
> I do not completely understand why do we have to check for file permissions
> in all the objects in branch of the tree that we are not trying to read
> from /write to.
> We could have checked file permission on the ancestor that exists and if it
> matches what we expect, the return true.
> Please confirm if this is a bug so that I can submit a patch else let me know
> what I am missing ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14234) TestHiveMetaStorePartitionSpecs does not drop database created in this test causes other test failure
[
https://issues.apache.org/jira/browse/HIVE-14234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378763#comment-15378763
]
niklaus xiao commented on HIVE-14234:
-
Tests failure not related, test exception as following:
{code}
Exception in thread "main" java.lang.RuntimeException: 404 Not Found
at
org.apache.hive.ptest.api.client.PTestClient.downloadTestResults(PTestClient.java:181)
at
org.apache.hive.ptest.api.client.PTestClient.testStart(PTestClient.java:129)
at
org.apache.hive.ptest.api.client.PTestClient.main(PTestClient.java:312)
{code}
> TestHiveMetaStorePartitionSpecs does not drop database created in this test
> causes other test failure
> -
>
> Key: HIVE-14234
> URL: https://issues.apache.org/jira/browse/HIVE-14234
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.3.0, 2.1.0
>Reporter: niklaus xiao
>Assignee: Mithun Radhakrishnan
>Priority: Minor
> Fix For: 1.3.0
>
> Attachments: HIVE-14234.patch
>
>
> TestHiveMetaStorePartitionSpecs creates a database named
> testpartitionspecs_db, but never drop it, sometimes causes
> TestObjectStore#testDatabaseOps failed:
> {code}
> testDatabaseOps(org.apache.hadoop.hive.metastore.TestObjectStore) Time
> elapsed: 0.188 sec <<< FAILURE!
> java.lang.AssertionError: expected:<2> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps(TestObjectStore.java:120)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14216) CREATE TABLE LIKE doesn't copy some attributes
[ https://issues.apache.org/jira/browse/HIVE-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376243#comment-15376243 ] niklaus xiao commented on HIVE-14216: - Fixed by this https://issues.apache.org/jira/browse/HIVE-10771 > CREATE TABLE LIKE doesn't copy some attributes > -- > > Key: HIVE-14216 > URL: https://issues.apache.org/jira/browse/HIVE-14216 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 > Environment: Hive 1.1 > Hadoop 2.6 >Reporter: Ruslan Dautkhanov >Priority: Critical > > CREATE TABLE LIKE doesn't copy some attributes, like skip.header.line.count > We use CREATE TABLE LIKE to create tables from a template table. > We have to do re-apply skip.header.line.count=1 every time to the new table, > although template table has skip.header.line.count set to 1, CREATE TABLE > LIKE does not carry it over to the new table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14234) TestHiveMetaStorePartitionSpecs does not drop database created in this test causes other test failure
[
https://issues.apache.org/jira/browse/HIVE-14234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14234:
Assignee: Mithun Radhakrishnan (was: niklaus xiao)
> TestHiveMetaStorePartitionSpecs does not drop database created in this test
> causes other test failure
> -
>
> Key: HIVE-14234
> URL: https://issues.apache.org/jira/browse/HIVE-14234
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.3.0, 2.1.0
>Reporter: niklaus xiao
>Assignee: Mithun Radhakrishnan
>Priority: Minor
> Fix For: 1.3.0
>
> Attachments: HIVE-14234.patch
>
>
> TestHiveMetaStorePartitionSpecs creates a database named
> testpartitionspecs_db, but never drop it, sometimes causes
> TestObjectStore#testDatabaseOps failed:
> {code}
> testDatabaseOps(org.apache.hadoop.hive.metastore.TestObjectStore) Time
> elapsed: 0.188 sec <<< FAILURE!
> java.lang.AssertionError: expected:<2> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps(TestObjectStore.java:120)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14234) TestHiveMetaStorePartitionSpecs does not drop database created in this test causes other test failure
[
https://issues.apache.org/jira/browse/HIVE-14234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376113#comment-15376113
]
niklaus xiao commented on HIVE-14234:
-
cc [~alangates]
> TestHiveMetaStorePartitionSpecs does not drop database created in this test
> causes other test failure
> -
>
> Key: HIVE-14234
> URL: https://issues.apache.org/jira/browse/HIVE-14234
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.3.0, 2.1.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
>Priority: Minor
> Fix For: 1.3.0
>
> Attachments: HIVE-14234.patch
>
>
> TestHiveMetaStorePartitionSpecs creates a database named
> testpartitionspecs_db, but never drop it, sometimes causes
> TestObjectStore#testDatabaseOps failed:
> {code}
> testDatabaseOps(org.apache.hadoop.hive.metastore.TestObjectStore) Time
> elapsed: 0.188 sec <<< FAILURE!
> java.lang.AssertionError: expected:<2> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps(TestObjectStore.java:120)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14234) TestHiveMetaStorePartitionSpecs does not drop database created in this test causes other test failure
[
https://issues.apache.org/jira/browse/HIVE-14234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376111#comment-15376111
]
niklaus xiao commented on HIVE-14234:
-
Small patch.
Can you take a look, since you are the origin author of this code. [~mithun]
> TestHiveMetaStorePartitionSpecs does not drop database created in this test
> causes other test failure
> -
>
> Key: HIVE-14234
> URL: https://issues.apache.org/jira/browse/HIVE-14234
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.3.0, 2.1.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
>Priority: Minor
> Fix For: 1.3.0
>
> Attachments: HIVE-14234.patch
>
>
> TestHiveMetaStorePartitionSpecs creates a database named
> testpartitionspecs_db, but never drop it, sometimes causes
> TestObjectStore#testDatabaseOps failed:
> {code}
> testDatabaseOps(org.apache.hadoop.hive.metastore.TestObjectStore) Time
> elapsed: 0.188 sec <<< FAILURE!
> java.lang.AssertionError: expected:<2> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps(TestObjectStore.java:120)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14234) TestHiveMetaStorePartitionSpecs does not drop database created in this test causes other test failure
[
https://issues.apache.org/jira/browse/HIVE-14234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14234:
Attachment: HIVE-14234.patch
> TestHiveMetaStorePartitionSpecs does not drop database created in this test
> causes other test failure
> -
>
> Key: HIVE-14234
> URL: https://issues.apache.org/jira/browse/HIVE-14234
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.3.0, 2.1.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
>Priority: Minor
> Fix For: 1.3.0
>
> Attachments: HIVE-14234.patch
>
>
> TestHiveMetaStorePartitionSpecs creates a database named
> testpartitionspecs_db, but never drop it, sometimes causes
> TestObjectStore#testDatabaseOps failed:
> {code}
> testDatabaseOps(org.apache.hadoop.hive.metastore.TestObjectStore) Time
> elapsed: 0.188 sec <<< FAILURE!
> java.lang.AssertionError: expected:<2> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps(TestObjectStore.java:120)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14234) TestHiveMetaStorePartitionSpecs does not drop database created in this test causes other test failure
[
https://issues.apache.org/jira/browse/HIVE-14234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14234:
Fix Version/s: 1.3.0
Status: Patch Available (was: Open)
> TestHiveMetaStorePartitionSpecs does not drop database created in this test
> causes other test failure
> -
>
> Key: HIVE-14234
> URL: https://issues.apache.org/jira/browse/HIVE-14234
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 2.1.0, 1.3.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
>Priority: Minor
> Fix For: 1.3.0
>
>
> TestHiveMetaStorePartitionSpecs creates a database named
> testpartitionspecs_db, but never drop it, sometimes causes
> TestObjectStore#testDatabaseOps failed:
> {code}
> testDatabaseOps(org.apache.hadoop.hive.metastore.TestObjectStore) Time
> elapsed: 0.188 sec <<< FAILURE!
> java.lang.AssertionError: expected:<2> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps(TestObjectStore.java:120)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10875) Select query with view in subquery adds underlying table as direct input
[
https://issues.apache.org/jira/browse/HIVE-10875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374215#comment-15374215
]
niklaus xiao commented on HIVE-10875:
-
Seems this query has the same issue
{code}
select * from V union all select * from V;
{/code}
[~thejas] Can you take a look?
> Select query with view in subquery adds underlying table as direct input
>
>
> Key: HIVE-10875
> URL: https://issues.apache.org/jira/browse/HIVE-10875
> Project: Hive
> Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 1.2.1
>
> Attachments: HIVE-10875.1.patch, HIVE-10875.2.patch
>
>
> In the following case,
> {code}
> create view V as select * from T;
> select * from (select * from V) A;
> {code}
> The semantic analyzer inputs contain input table T as a direct input instead
> of adding it as an indirect input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-10875) Select query with view in subquery adds underlying table as direct input
[
https://issues.apache.org/jira/browse/HIVE-10875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374215#comment-15374215
]
niklaus xiao edited comment on HIVE-10875 at 7/13/16 2:36 AM:
--
Seems this query has the same issue
{code}
select * from V union all select * from V;
{code}
[~thejas] Can you take a look?
was (Author: niklaus.xiao):
Seems this query has the same issue
{code}
select * from V union all select * from V;
{/code}
[~thejas] Can you take a look?
> Select query with view in subquery adds underlying table as direct input
>
>
> Key: HIVE-10875
> URL: https://issues.apache.org/jira/browse/HIVE-10875
> Project: Hive
> Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 1.2.1
>
> Attachments: HIVE-10875.1.patch, HIVE-10875.2.patch
>
>
> In the following case,
> {code}
> create view V as select * from T;
> select * from (select * from V) A;
> {code}
> The semantic analyzer inputs contain input table T as a direct input instead
> of adding it as an indirect input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14145) Too small length of column 'PARAM_VALUE' in table 'SERDE_PARAMS'
[
https://issues.apache.org/jira/browse/HIVE-14145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15363560#comment-15363560
]
niklaus xiao commented on HIVE-14145:
-
Met the same issue, can anyone review the patch ?
> Too small length of column 'PARAM_VALUE' in table 'SERDE_PARAMS'
>
>
> Key: HIVE-14145
> URL: https://issues.apache.org/jira/browse/HIVE-14145
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
> Fix For: 2.1.1
>
> Attachments: HIVE-14145.1.patch, HIVE-14145.2.patch
>
>
> Customer has following table
> {code}
> create external table hive_hbase_test(
> HBASE_KEY string,
> ENTITY_NAME string,
> ENTITY_ID string,
> CLAIM_HEADER_ID string,
> CLAIM_LINE_ID string,
> MEDICAL_CLAIM_SOURCE_SYSTEM string,
> UNIQUE_MEMBER_ID string,
> MEMBER_SOURCE_SYSTEM string,
> SUBSCRIBER_ID string,
> COVERAGE_CLASS_CODE string,
> SERVICING_PROVIDER_ID string,
> PROVIDER_SOURCE_SYSTEM string,
> SERVICING_PROVIDER_SPECIALTY string,
> SERVICING_STANDARD_PROVIDER_SPECIALTY string,
> SERVICING_PROVIDER_TYPE_CODE string,
> REFERRING_PROVIDER_ID string,
> ADMITTING_PROVIDER_ID string,
> ATTENDING_PROVIDER_ID string,
> OPERATING_PROVIDER_ID string,
> BILLING_PROVIDER_ID string,
> ORDERING_PROVIDER_ID string,
> HEALTH_PLAN_SOURCE_ID string,
> HEALTH_PLAN_PAYER_NAME string,
> BUSINESS_UNIT string,
> OPERATING_UNIT string,
> PRODUCT string,
> MARKET string,
> DEPARTMENT string,
> IPA string,
> SUPPLEMENTAL_DATA_TYPE string,
> PSEUDO_CLAIM_FLAG string,
> CLAIM_STATUS string,
> CLAIM_LINE_STATUS string,
> CLAIM_DENIED_FLAG string,
> SERVICE_LINE_DENIED_FLAG string,
> DENIED_REASON_CODE string,
> SERVICE_LINE_DENIED_REASON_CODE string,
> DAYS_DENIED int,
> DIAGNOSIS_DATE timestamp,
> SERVICE_DATE TIMESTAMP,
> SERVICE_FROM_DATE TIMESTAMP,
> SERVICE_TO_DATE TIMESTAMP,
> ADMIT_DATE TIMESTAMP,
> ADMIT_TYPE string,
> ADMIT_SOURCE_TYPE string,
> DISCHARGE_DATE TIMESTAMP,
> DISCHARGE_STATUS_CODE string,
> SERVICE_LINE_TYPE_OF_SERVICE string,
> TYPE_OF_BILL_CODE string,
> INPATIENT_FLAG string,
> PLACE_OF_SERVICE_CODE string,
> FACILITY_CODE string,
> AUTHORIZATION_NUMBER string,
> CLAIM_REFERRAL_NUMBER string,
> CLAIM_TYPE string,
> CLAIM_ADJUSTMENT_TYPE string,
> ICD_DIAGNOSIS_CODE_1 string,
> PRESENT_ON_ADMISSION_FLAG_1 string,
> ICD_DIAGNOSIS_CODE_2 string,
> PRESENT_ON_ADMISSION_FLAG_2 string,
> ICD_DIAGNOSIS_CODE_3 string,
> PRESENT_ON_ADMISSION_FLAG_3 string,
> ICD_DIAGNOSIS_CODE_4 string,
> PRESENT_ON_ADMISSION_FLAG_4 string,
> ICD_DIAGNOSIS_CODE_5 string,
> PRESENT_ON_ADMISSION_FLAG_5 string,
> ICD_DIAGNOSIS_CODE_6 string,
> PRESENT_ON_ADMISSION_FLAG_6 string,
> ICD_DIAGNOSIS_CODE_7 string,
> PRESENT_ON_ADMISSION_FLAG_7 string,
> ICD_DIAGNOSIS_CODE_8 string,
> PRESENT_ON_ADMISSION_FLAG_8 string,
> ICD_DIAGNOSIS_CODE_9 string,
> PRESENT_ON_ADMISSION_FLAG_9 string,
> ICD_DIAGNOSIS_CODE_10 string,
> PRESENT_ON_ADMISSION_FLAG_10 string,
> ICD_DIAGNOSIS_CODE_11 string,
> PRESENT_ON_ADMISSION_FLAG_11 string,
> ICD_DIAGNOSIS_CODE_12 string,
> PRESENT_ON_ADMISSION_FLAG_12 string,
> ICD_DIAGNOSIS_CODE_13 string,
> PRESENT_ON_ADMISSION_FLAG_13 string,
> ICD_DIAGNOSIS_CODE_14 string,
> PRESENT_ON_ADMISSION_FLAG_14 string,
> ICD_DIAGNOSIS_CODE_15 string,
> PRESENT_ON_ADMISSION_FLAG_15 string,
> ICD_DIAGNOSIS_CODE_16 string,
> PRESENT_ON_ADMISSION_FLAG_16 string,
> ICD_DIAGNOSIS_CODE_17 string,
> PRESENT_ON_ADMISSION_FLAG_17 string,
> ICD_DIAGNOSIS_CODE_18 string,
> PRESENT_ON_ADMISSION_FLAG_18 string,
> ICD_DIAGNOSIS_CODE_19 string,
> PRESENT_ON_ADMISSION_FLAG_19 string,
> ICD_DIAGNOSIS_CODE_20 string,
> PRESENT_ON_ADMISSION_FLAG_20 string,
> ICD_DIAGNOSIS_CODE_21 string,
> PRESENT_ON_ADMISSION_FLAG_21 string,
> ICD_DIAGNOSIS_CODE_22 string,
> PRESENT_ON_ADMISSION_FLAG_22 string,
> ICD_DIAGNOSIS_CODE_23 string,
> PRESENT_ON_ADMISSION_FLAG_23 string,
> ICD_DIAGNOSIS_CODE_24 string,
> PRESENT_ON_ADMISSION_FLAG_24 string,
> ICD_DIAGNOSIS_CODE_25 string,
> PRESENT_ON_ADMISSION_FLAG_25 string,
> QUANTITY_OF_SERVICES decimal(10,2),
> REVENUE_CODE string,
> PROCEDURE_CODE string,
> PROCEDURE_CODE_MODIFIER_1 string,
> PROCEDURE_CODE_MODIFIER_2 string,
> PROCEDURE_CODE_MODIFIER_3 string,
> PROCEDURE_CODE_MODIFIER_4 string,
> ICD_VERSION_CODE_TYPE string,
> ICD_PROCEDURE_CODE_1 string,
> ICD_PROCEDURE_CODE_2 string,
> ICD_PROCEDURE_CODE_3 string,
> ICD_PROCEDURE_CODE_4 string,
> ICD_PROCEDURE_CODE_5 string,
> ICD_PROCEDURE_CODE_6 string,
> ICD_PROCEDURE_CODE_7 string,
> ICD_PROCEDURE_CODE_8 string,
> ICD_PROCEDURE_CODE_9 string,
> ICD_PROCEDURE_CODE_10 string,
> ICD_PROCEDURE_CODE_11 string,
> ICD_PROCEDURE_CODE_12 string,
> ICD_PROCEDURE_CODE_13 string,
> ICD_PROCEDURE_CODE_14 string,
> ICD_PROCEDURE_CODE_15 string,
> ICD_PR
[jira] [Commented] (HIVE-14081) Appending a variable value into the hive query inside java code gives me an error
[
https://issues.apache.org/jira/browse/HIVE-14081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15363552#comment-15363552
]
niklaus xiao commented on HIVE-14081:
-
Try this:
{code}
res=stm.executeQuery("select * from dataset where c_name =
'"+jComboBox1.getSelectedItem() + "'");
{code}
> Appending a variable value into the hive query inside java code gives me an
> error
> -
>
> Key: HIVE-14081
> URL: https://issues.apache.org/jira/browse/HIVE-14081
> Project: Hive
> Issue Type: Bug
> Components: API
>Affects Versions: 0.13.0
>Reporter: Amey D
>
> New to this forum please help or guide me where do i find the solution to
> this error.
> Query inside java :
> res=stm.executeQuery("select * from dataset where c_name =
> "+jComboBox1.getSelectedItem());
> Error :
> FAILED: SemanticException [Error 10004]: Line 1:35 Invalid table alias or
> column reference 'AAPL': (possible column names are:
> Cannot get over this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14156) Problem with Chinese characters as partition value when using MySQL
[
https://issues.apache.org/jira/browse/HIVE-14156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15362443#comment-15362443
]
niklaus xiao edited comment on HIVE-14156 at 7/5/16 12:53 PM:
--
I tried this on postgres, seems not an issue.
{code}
create table foo (name string, age int) partitioned by (city string) row format
delimited fields terminated by ',';
alter table foo add partition(city='深圳');
show partitions foo;
++--+
| partition |
++--+
| city=深圳|
++--+
1 row selected (0.355 seconds)
{code}
was (Author: niklaus.xiao):
I tried this on postgres, seems not an issue.
{quote}
create table foo (name string, age int) partitioned by (city string) row format
delimited fields terminated by ',';
alter table foo add partition(city='深圳');
show partitions foo;
++--+
| partition |
++--+
| city=深圳|
++--+
1 row selected (0.355 seconds)
{quote}
> Problem with Chinese characters as partition value when using MySQL
> ---
>
> Key: HIVE-14156
> URL: https://issues.apache.org/jira/browse/HIVE-14156
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Bing Li
>Assignee: Bing Li
>
> Steps to reproduce:
> create table t1 (name string, age int) partitioned by (city string) row
> format delimited fields terminated by ',';
> load data local inpath '/tmp/chn-partition.txt' overwrite into table t1
> partition (city='北京');
> The content of /tmp/chn-partition.txt:
> 小明,20
> 小红,15
> 张三,36
> 李四,50
> When check the partition value in MySQL, it shows ?? instead of "北京".
> When run "drop table t1", it will hang.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14146) Column comments with "\n" character "corrupts" table metadata
[
https://issues.apache.org/jira/browse/HIVE-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15362446#comment-15362446
]
niklaus xiao commented on HIVE-14146:
-
You can try this:
{code}
desc pretty commtest;
+--++--+
| col_name | data_type | comment |
+--++--+--+
| col_name data_type comment | NULL | NULL |
| | NULL | NULL |
| first_nm stringIndicates First name | NULL | NULL |
|of an individual | NULL | NULL |
+--++--+
{code}
> Column comments with "\n" character "corrupts" table metadata
> -
>
> Key: HIVE-14146
> URL: https://issues.apache.org/jira/browse/HIVE-14146
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>
> Create a table with the following(noting the \n in the COMMENT):
> {noformat}
> CREATE TABLE commtest(first_nm string COMMENT 'Indicates First name\nof an
> individual’);
> {noformat}
> Describe shows that now the metadata is messed up:
> {noformat}
> beeline> describe commtest;
> +---++---+--+
> | col_name | data_type |comment|
> +---++---+--+
> | first_nm | string | Indicates First name |
> | of an individual | NULL | NULL |
> +---++---+--+
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14156) Problem with Chinese characters as partition value when using MySQL
[
https://issues.apache.org/jira/browse/HIVE-14156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15362443#comment-15362443
]
niklaus xiao commented on HIVE-14156:
-
I tried this on postgres, seems not an issue.
{quote}
create table foo (name string, age int) partitioned by (city string) row format
delimited fields terminated by ',';
alter table foo add partition(city='深圳');
show partitions foo;
++--+
| partition |
++--+
| city=深圳|
++--+
1 row selected (0.355 seconds)
{quote}
> Problem with Chinese characters as partition value when using MySQL
> ---
>
> Key: HIVE-14156
> URL: https://issues.apache.org/jira/browse/HIVE-14156
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Bing Li
>Assignee: Bing Li
>
> Steps to reproduce:
> create table t1 (name string, age int) partitioned by (city string) row
> format delimited fields terminated by ',';
> load data local inpath '/tmp/chn-partition.txt' overwrite into table t1
> partition (city='北京');
> The content of /tmp/chn-partition.txt:
> 小明,20
> 小红,15
> 张三,36
> 李四,50
> When check the partition value in MySQL, it shows ?? instead of "北京".
> When run "drop table t1", it will hang.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-12154) Load data inpath 'PATTERN' into table should only check files match the PATTERN
[
https://issues.apache.org/jira/browse/HIVE-12154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-12154:
Comment: was deleted
(was: Use
{quote}
fs.globStatus(pattern);
{quote}
instead of
{quote}
fs.listStatus(path);
{quote}
Attached the initial patch.)
> Load data inpath 'PATTERN' into table should only check files match the
> PATTERN
> ---
>
> Key: HIVE-12154
> URL: https://issues.apache.org/jira/browse/HIVE-12154
> Project: Hive
> Issue Type: Bug
> Components: SQLStandardAuthorization
>Affects Versions: 0.13.1, 1.0.0, 1.2.0, 1.1.0, 1.2.1
>Reporter: niklaus xiao
>Priority: Minor
>
> We are using flume to sink data to HDFS directory '/tmp/test/', temporal
> files that flume actively writes into has a suffix .tmp, after writes finish,
> the file will be renamed to SAMPLE.data.
> Hive periodic task execute script like
> {quote}
> load data inpath '/tmp/test/*.data' into table t1;
> {quote}
> This exception happens sometimes
> {quote}
> 2015-10-12 19:38:00,133 | ERROR | HiveServer2-Handler-Pool: Thread-57 |
> FAILED: HiveAuthzPluginException Error getting permissions for
> hdfs://hacluster/tmp/test/*.data: null
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzPluginException:
> Error getting permissions for hdfs://hacluster/tmp/test/*.data: null
> ...
> Caused by: java.io.FileNotFoundException: Path not found
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAccess(FSNamesystem.java:8175)
> {quote}
> I digged into the code, and found that SQLStdHiveAuthorizationValidator
> checks all the files in /tmp/test/ directory, but when checks the permission
> of .tmp file, the file is renamed to .data, hdfs cannot find this file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13883) WebHCat leaves token crc file never gets deleted
[
https://issues.apache.org/jira/browse/HIVE-13883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15362428#comment-15362428
]
niklaus xiao commented on HIVE-13883:
-
[~sushanth] Could you take a look?
> WebHCat leaves token crc file never gets deleted
>
>
> Key: HIVE-13883
> URL: https://issues.apache.org/jira/browse/HIVE-13883
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Affects Versions: 1.2.0, 1.1.1, 1.2.1, 2.0.1
>Reporter: niklaus xiao
>Priority: Minor
> Attachments: HIVE-13883.patch
>
>
> In one of our long run environment, there are thousands of
> /tmp/.templeton*.tmp.crc files,
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc
> ...
> -rw-r--r-- 1 omm wheel 12 May 26 18:15
> /tmp/.templeton6676048390600607654.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:14
> /tmp/.templeton2733383617337556503.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:12
> /tmp/.templeton2183121761801669064.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:11
> /tmp/.templeton2689764046140543879.tmp.crc
> ...
> {quote}
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc | wc -l
> 17986
> {quote}
> It's created by webhcat,
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L193]
> and never gets deleted
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L110]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14113) Create function failed but function in show function list
[
https://issues.apache.org/jira/browse/HIVE-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15362403#comment-15362403
]
niklaus xiao commented on HIVE-14113:
-
test failure unrelated.
> Create function failed but function in show function list
> -
>
> Key: HIVE-14113
> URL: https://issues.apache.org/jira/browse/HIVE-14113
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.2.0
>Reporter: niklaus xiao
>Assignee: Navis
> Fix For: 1.3.0
>
> Attachments: HIVE-14113.1.patch
>
>
> 1. create function with invalid hdfs path, /udf/udf-test.jar does not exists
> {quote}
> create function my_lower as 'com.tang.UDFLower' using jar
> 'hdfs:///udf/udf-test.jar';
> {quote}
> Failed with following exception:
> {quote}
> 0: jdbc:hive2://189.39.151.44:1/> create function my_lower as
> 'com.tang.UDFLower' using jar 'hdfs:///udf/udf-test.jar';
> INFO : converting to local hdfs:///udf/udf-test.jar
> ERROR : Failed to read external resource hdfs:///udf/udf-test.jar
> java.lang.RuntimeException: Failed to read external resource
> hdfs:///udf/udf-test.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1384)
> at
> org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1340)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1264)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1250)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:306)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:466)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerPermanentFunction(Registry.java:206)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerPermanentFunction(FunctionRegistry.java:1551)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.createPermanentFunction(FunctionTask.java:136)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:75)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1965)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1723)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1475)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1283)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:167)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:258)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: File does not exist:
> hdfs:/udf/udf-test.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1391)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1383)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2034)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2003)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1979)
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1370)
> ... 28 more
> ERROR : Failed to register default.my_lower using class com.tang.UDFLower
> Error: Error while processing statement: FAILED: Executio
