[jira] [Created] (HIVE-20347) hive.optimize.sort.dynamic.partition should work with partitioned CTAS and MV
Jesus Camacho Rodriguez created HIVE-20347: -- Summary: hive.optimize.sort.dynamic.partition should work with partitioned CTAS and MV Key: HIVE-20347 URL: https://issues.apache.org/jira/browse/HIVE-20347 Project: Hive Issue Type: Bug Components: Materialized views Affects Versions: 4.0.0, 3.2.0 Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20346) [Hive Upgrade] A rollback script is necessary when upgrade hive
wuchang created HIVE-20346: -- Summary: [Hive Upgrade] A rollback script is necessary when upgrade hive Key: HIVE-20346 URL: https://issues.apache.org/jira/browse/HIVE-20346 Project: Hive Issue Type: Task Reporter: wuchang When we upgrade hive from a low version(1.1.0) to higher version(2.3.3), we will have to: # backup the old hive metastore database # Use schema tool to upgrade the hive metastore database schema. Then several days after the upgrading, new version hive have some critical problems and we want to revert to previous hive, what could we do? Since during these days the data in hive metastore has changed. So, if we directly switch to the old hive metastore, any metadata changes after the upgrading will lose. So , I think we need a rollback script, which is used to downgrade our high version metastore schema to low version schema. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20345) Drop database may hang by the change in HIVE-11258
Aihua Xu created HIVE-20345:
---
Summary: Drop database may hang by the change in HIVE-11258
Key: HIVE-20345
URL: https://issues.apache.org/jira/browse/HIVE-20345
Project: Hive
Issue Type: Bug
Components: Standalone Metastore
Affects Versions: 2.0.0, 1.3.0
Reporter: Aihua Xu
Assignee: Aihua Xu
In HiveMetaStore.java drop_database_core function, HIVE-11258 updates the
startIndex from endIndex incorrectly inside {{if (tables != null &&
!tables.isEmpty())}} statement. If the tables get deleted before
getTableObjectsByName() call, then returned table list is empty and startIndex
won't get updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Review Request 68272: HIVE-20336
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68272/ --- Review request for hive and Ashutosh Chauhan. Bugs: HIVE-20336 https://issues.apache.org/jira/browse/HIVE-20336 Repository: hive-git Description --- HIVE-20336 Diffs - ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 37bc153631f9cee415338c53a27ec396781caca2 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 474c793ec3961dddab866fe5f185a26233b082cf ql/src/java/org/apache/hadoop/hive/ql/parse/TableMask.java ee93cf65fd3759a175a188e3545ebbce79954281 ql/src/test/queries/clientnegative/masking_mv.q PRE-CREATION ql/src/test/queries/clientpositive/masking_mv.q PRE-CREATION ql/src/test/results/clientnegative/masking_mv.q.out PRE-CREATION ql/src/test/results/clientpositive/masking_mv.q.out PRE-CREATION Diff: https://reviews.apache.org/r/68272/diff/1/ Testing --- Thanks, Jesús Camacho Rodríguez
[jira] [Created] (HIVE-20344) PrivilegeSynchronizer for SBA might hit AccessControlException
Daniel Dai created HIVE-20344:
-
Summary: PrivilegeSynchronizer for SBA might hit
AccessControlException
Key: HIVE-20344
URL: https://issues.apache.org/jira/browse/HIVE-20344
Project: Hive
Issue Type: Improvement
Reporter: Daniel Dai
Assignee: Daniel Dai
If "hive" user does not have privilege of corresponding hdfs folders,
PrivilegeSynchronizer won't be able to get metadata of the table because SBA is
preventing it. Here is a sample stack:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.security.AccessControlException: Permission denied: user=hive,
access=EXECUTE, inode="/tmp/sba_is/sba_db":hrt_7:hrt_qa:dr
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:315)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:242)
at
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:512)
at
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:305)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1850)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1834)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:1784)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAccess(FSNamesystem.java:7767)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.checkAccess(NameNodeRpcServer.java:2217)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.checkAccess(ClientNamenodeProtocolServerSideTranslatorPB.java:1659)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
at
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(StorageBasedAuthorizationProvider.java:424)
at
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(StorageBasedAuthorizationProvider.java:382)
at
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.authorize(StorageBasedAuthorizationProvider.java:355)
at
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.authorize(StorageBasedAuthorizationProvider.java:203)
at
org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener.authorizeReadTable(AuthorizationPreEventListener.java:192)
... 23 more
{code}
I simply skip the table if that happens. In practice, managed tables are owned
by "hive" user, so only external tables will be impacted. User need to grant
execute permission of db folder and read permission of the table folders to
"hive" user if they want to query the information schema for the tables, whose
permission is only granted via SBA.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
Rajkumar Singh created HIVE-20343:
-
Summary: Hive 3: CTAS does not respect transactional_properties
Key: HIVE-20343
URL: https://issues.apache.org/jira/browse/HIVE-20343
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 3.1.0
Environment: hive-3
Reporter: Rajkumar Singh
Steps to reproduce:
{code}
create table ctasexampleinsertonly stored as orc TBLPROPERTIES
("transactional_properties"="insert_only") as select * from testtable limit 1;
describe formatted ctasexampleinsertonly
+---++---+
| col_name| data_type
|comment|
+---++---+
| # col_name| data_type
| comment |
| name | varchar(8)
| |
| time | double
| |
| | NULL
| NULL |
| # Detailed Table Information | NULL
| NULL |
| Database: | default
| NULL |
| OwnerType:| USER
| NULL |
| Owner:| hive
| NULL |
| CreateTime: | Wed Aug 08 21:35:15 UTC 2018
| NULL |
| LastAccessTime: | UNKNOWN
| NULL |
| Retention:| 0
| NULL |
| Location: |
hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
| NULL |
| Table Type: | MANAGED_TABLE
| NULL |
| Table Parameters: | NULL
| NULL |
| | COLUMN_STATS_ACCURATE
| {}|
| | bucketing_version
| 2 |
| | numFiles
| 1 |
| | numRows
| 1 |
| | rawDataSize
| 0 |
| | totalSize
| 754 |
| | transactional
| true |
| | transactional_properties
| default |
| | transient_lastDdlTime
| 1533764115|
| | NULL
| NULL |
| # Storage Information | NULL
| NULL |
| SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde
| NULL |
| InputFormat: |
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL |
| OutputFormat: |
org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL |
| Compressed: | No
| NULL |
| Num Buckets: | -1
| NULL |
| Bucket Columns: | []
| NULL |
| Sort Columns: | []
| NULL |
| Storage Desc Params: | NULL
| NULL |
| | serialization.format
| 1 |
+---++---+
{code}
this creates a problem with insert
{code}
CREATE TABLE
[jira] [Created] (HIVE-20342) Create table from CTAS with location fails for managed tables
Pablo Idiaquez created HIVE-20342:
-
Summary: Create table from CTAS with location fails for managed
tables
Key: HIVE-20342
URL: https://issues.apache.org/jira/browse/HIVE-20342
Project: Hive
Issue Type: Bug
Components: Beeline, HiveServer2
Affects Versions: 3.0.0
Reporter: Pablo Idiaquez
When attempting to create table from SELECT with LOCATION (managed tables)
To reproduce CREATE source table CLASS with some data
then use the code below to CREATE table TABLE42 from select on CLASS and
LOCATION /tmp/test1
{code:java}
CREATE TABLE TABLE42 ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe' STORED AS RCFILE
LOCATION '/tmp/test1'
AS SELECT * FROM
CLASS;
{code}
it fails with ERROR:
{code:java}
ERROR : Job Commit failed with exception
'org.apache.hadoop.hive.ql.metadata.HiveException(The following files were
committed but not found: [/tmp/test1/delta_001_001_/00_0])'
org.apache.hadoop.hive.ql.metadata.HiveException: The following files were
committed but not found: [/tmp/test1/delta_001_001_/00_0]
at
org.apache.hadoop.hive.ql.exec.Utilities.handleMmTableFinalPath(Utilities.java:4329)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1393)
{code}
If the same is attempted for an EXTERNAL table will work OK
{code:java}
CREATE EXTERNAL TABLE test3 ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe' STORED AS RCFILE
LOCATION '/tmp/test2' AS SELECT * FROM test;
{code}
if we CREATE table on statement 1, and INSERT from SELECT on statement 2 , will
work OK.
(Step1 CREATE TABLE)
{code:java}
CREATE TABLE `TABLE42`(
`COL1` double,
`COL2` varchar(8),
`COL3` varchar(1),
`COL4` double,
`COL5` double)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.RCFileInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.RCFileOutputFormat'
LOCATION '/tmp/test1';
{code}
(Step2 INSERT from SELECT)
{code:java}
INSERT into TABLE42 select * FROM CLASS;
{code}
will work as expected.
Thanks
Pablo
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-20341) Druid Needs Explicit CASTs from Timestamp to STRING when the output of timestamp function is used as String
Nishant Bangarwa created HIVE-20341:
---
Summary: Druid Needs Explicit CASTs from Timestamp to STRING when
the output of timestamp function is used as String
Key: HIVE-20341
URL: https://issues.apache.org/jira/browse/HIVE-20341
Project: Hive
Issue Type: Bug
Reporter: Nishant Bangarwa
Druid timestamp expression functions returns numeric values in form of millis
since epoch.
Functions that use the output of the timestamp functions as String return
different values for tables stored in HIVE and Druid.
{code}
SELECT SUBSTRING(to_date(datetime0),4) FROM tableau_orc.calcs;
| 4-07-25 |
SELECT SUBSTRING(to_date(datetime0),4) FROM druid_tableau.calcs;
| 002240 |
SELECT CONCAT(to_date(datetime0),' 00:00:00') FROM tableau_orc.calcs;
| 2004-07-17 00:00:00 |
SELECT CONCAT(to_date(datetime0),' 00:00:00') FROM druid_tableau.calcs;
| 109045440 00:00:00 |
{code}
We need to add explicit CAST to String before generating Druid expressions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-20340) Druid Needs Explicit CASTs from Timestamp to STRING when the output of timestamp function is used as String
Vineet Garg created HIVE-20340:
--
Summary: Druid Needs Explicit CASTs from Timestamp to STRING when
the output of timestamp function is used as String
Key: HIVE-20340
URL: https://issues.apache.org/jira/browse/HIVE-20340
Project: Hive
Issue Type: Bug
Components: Druid integration
Reporter: Vineet Garg
Assignee: Vineet Garg
Druid time expressions return numeric values in form of ms (instead of
formatted timestamp). Because of this expressions/function which expects its
argument as string type ended up returning different values for time
expressions input.
e.g.
{code}
SELECT SUBSTRING(to_date(datetime0),4) FROM tableau_orc.calcs;
| 4-07-25 |
SELECT SUBSTRING(to_date(datetime0),4) FROM druid_tableau.calcs;
| 002240 |
SELECT CONCAT(to_date(datetime0),' 00:00:00') FROM tableau_orc.calcs;
| 2004-07-17 00:00:00 |
SELECT CONCAT(to_date(datetime0),' 00:00:00') FROM druid_tableau.calcs;
| 109045440 00:00:00 |
{code}
Druid needs explicit cast to make this work
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-20339) Vectorization: Lift unneeded restriction causing some PTF with RANK not to be vectorized
Matt McCline created HIVE-20339: --- Summary: Vectorization: Lift unneeded restriction causing some PTF with RANK not to be vectorized Key: HIVE-20339 URL: https://issues.apache.org/jira/browse/HIVE-20339 Project: Hive Issue Type: Bug Components: Hive Reporter: Matt McCline Assignee: Matt McCline Unnecessary: "PTF operator: More than 1 argument expression of aggregation function rank" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20338) LLAP: Force synthetic file-id for filesystems which have HDFS protocol impls with POSIX mutation semantics
Gopal V created HIVE-20338: -- Summary: LLAP: Force synthetic file-id for filesystems which have HDFS protocol impls with POSIX mutation semantics Key: HIVE-20338 URL: https://issues.apache.org/jira/browse/HIVE-20338 Project: Hive Issue Type: Bug Reporter: Gopal V HDFS protocol does not guarantee immutability of files - the synthetic file-id includes the mtime of the file as well, which is a fail-safe for such filesystems. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20337) CachedStore: getPartitionsByExpr is not populating the partition list correctly
Vaibhav Gumashta created HIVE-20337: --- Summary: CachedStore: getPartitionsByExpr is not populating the partition list correctly Key: HIVE-20337 URL: https://issues.apache.org/jira/browse/HIVE-20337 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 3.1.0 Reporter: Vaibhav Gumashta -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20336) Masking and filtering policies for materialized views
Jesus Camacho Rodriguez created HIVE-20336: -- Summary: Masking and filtering policies for materialized views Key: HIVE-20336 URL: https://issues.apache.org/jira/browse/HIVE-20336 Project: Hive Issue Type: Bug Components: Authorization, Materialized views Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Implement masking and filtering policies for materialized views. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 68261: HIVE-20332
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68261/ --- (Updated Aug. 8, 2018, 3:39 p.m.) Review request for hive and Ashutosh Chauhan. Bugs: HIVE-20332 https://issues.apache.org/jira/browse/HIVE-20332 Repository: hive-git Description --- HIVE-20332 Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5bdcac88d0015d2410da050524e6697a22d83eb9 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveDefaultRelMetadataProvider.java 635d27e723dc1d260574723296f3484c26106a9c ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveMaterializedViewsRelMetadataProvider.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java 43f8508ffbf4ba3cc46016e1d300d6ca9c2e8ccb ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdCumulativeCost.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdDistinctRowCount.java 80b939a9f65142baa149b79460b753ddf469aacf ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java 575902d78de2a7f95585c23a3c2fc03b9ce89478 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSize.java 97097381d9619e67bcab8a268d571d2a392485b3 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdUniqueKeys.java 3bf62c535cec1e7a3eac43f0ce40879dbfc89799 ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 361f150193a155d45eb64266f88eb88f0a881ad3 ql/src/test/results/clientpositive/llap/materialized_view_partitioned.q.out b12df11a98e55c00c8b77e8292666373f3509364 ql/src/test/results/clientpositive/llap/materialized_view_rebuild.q.out 4d37d82b6e1f3d4ab8b76c391fa94176356093c2 Diff: https://reviews.apache.org/r/68261/diff/2/ Changes: https://reviews.apache.org/r/68261/diff/1-2/ Testing --- Thanks, Jesús Camacho Rodríguez
