[jira] [Assigned] (HIVE-28512) CREATE TABLE x LIKE retain whitelisted table properties
[ https://issues.apache.org/jira/browse/HIVE-28512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala reassigned HIVE-28512: Assignee: Jintong Jiang (was: Sai Hemanth Gantasala) > CREATE TABLE x LIKE retain whitelisted table properties > --- > > Key: HIVE-28512 > URL: https://issues.apache.org/jira/browse/HIVE-28512 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Sai Hemanth Gantasala >Assignee: Jintong Jiang >Priority: Major > Labels: pull-request-available > > It would be good to retain properties in > HiveConf.ConfVars.DDL_CTL_PARAMETERS_WHITELIST for CTLT query, as this is > particularly useful for avro base tables as the schema can evolve over time > and avro schema is mentioned in the avro.schema.url tblproperty. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-28511) Include column info in the HivePrivilegeObject for insert operation
[ https://issues.apache.org/jira/browse/HIVE-28511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-28511. -- Assignee: Jintong Jiang Resolution: Fixed Thanks for your contribution [~hazeljiang]. Patch has been merged into the master branch. > Include column info in the HivePrivilegeObject for insert operation > --- > > Key: HIVE-28511 > URL: https://issues.apache.org/jira/browse/HIVE-28511 > Project: Hive > Issue Type: Bug >Reporter: Jintong Jiang >Assignee: Jintong Jiang >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > > Enhance the HivePrivilegeObject by incorporating detailed column-level > information specifically for INSERT operations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28511) Include column info in the HivePrivilegeObject for insert operation
[ https://issues.apache.org/jira/browse/HIVE-28511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala updated HIVE-28511: - Fix Version/s: 4.1.0 > Include column info in the HivePrivilegeObject for insert operation > --- > > Key: HIVE-28511 > URL: https://issues.apache.org/jira/browse/HIVE-28511 > Project: Hive > Issue Type: Bug >Reporter: Jintong Jiang >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > > Enhance the HivePrivilegeObject by incorporating detailed column-level > information specifically for INSERT operations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28511) User allowed to insert data into a hive table when there is a deny policy on a table column
[ https://issues.apache.org/jira/browse/HIVE-28511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala updated HIVE-28511: - Summary: User allowed to insert data into a hive table when there is a deny policy on a table column (was: [Hive side work] - User allowed to insert data into a hive table when there is a deny policy on a table column) > User allowed to insert data into a hive table when there is a deny policy on > a table column > --- > > Key: HIVE-28511 > URL: https://issues.apache.org/jira/browse/HIVE-28511 > Project: Hive > Issue Type: Bug >Reporter: Jintong Jiang >Priority: Major > > Need to deny User write/read to specific column when deny policy exists. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28294) drop database cascade operation can skip client side filtering while fetching tables in db
[ https://issues.apache.org/jira/browse/HIVE-28294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876883#comment-17876883 ] Sai Hemanth Gantasala commented on HIVE-28294: -- Patch merged into the master branch. Thanks, [~dengzh] for the review. > drop database cascade operation can skip client side filtering while fetching > tables in db > -- > > Key: HIVE-28294 > URL: https://issues.apache.org/jira/browse/HIVE-28294 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > > Drop database cascade operation fetches all tables in the DB, while doing so > we perform client-side filtering on the tables. We can avoid client-side > filtering as we anyway authorize on the tables in the DB for the drop > operation. > Also, we need to add functions in the database for authorization before > dropping the database. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-28294) drop database cascade operation can skip client side filtering while fetching tables in db
[ https://issues.apache.org/jira/browse/HIVE-28294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-28294. -- Fix Version/s: 4.1.0 Resolution: Fixed > drop database cascade operation can skip client side filtering while fetching > tables in db > -- > > Key: HIVE-28294 > URL: https://issues.apache.org/jira/browse/HIVE-28294 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > > Drop database cascade operation fetches all tables in the DB, while doing so > we perform client-side filtering on the tables. We can avoid client-side > filtering as we anyway authorize on the tables in the DB for the drop > operation. > Also, we need to add functions in the database for authorization before > dropping the database. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28134) Improve SecureCmdDoAs
[ https://issues.apache.org/jira/browse/HIVE-28134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala updated HIVE-28134: - Affects Version/s: 4.0.0-alpha-1 > Improve SecureCmdDoAs > - > > Key: HIVE-28134 > URL: https://issues.apache.org/jira/browse/HIVE-28134 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0, 4.0.0-alpha-1 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: hive-4.0.1-merged, hive-4.0.1-must, > pull-request-available > Fix For: 4.1.0 > > > Improve the SecureCmdDoAs code -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28134) Improve SecureCmdDoAs
[ https://issues.apache.org/jira/browse/HIVE-28134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala updated HIVE-28134: - Affects Version/s: 3.1.0 > Improve SecureCmdDoAs > - > > Key: HIVE-28134 > URL: https://issues.apache.org/jira/browse/HIVE-28134 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: hive-4.0.1-merged, hive-4.0.1-must, > pull-request-available > Fix For: 4.1.0 > > > Improve the SecureCmdDoAs code -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28447) Remove the filtering around the tableExists
[ https://issues.apache.org/jira/browse/HIVE-28447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17874926#comment-17874926 ] Sai Hemanth Gantasala commented on HIVE-28447: -- cc [~danielzhu] Can you work on this? > Remove the filtering around the tableExists > --- > > Key: HIVE-28447 > URL: https://issues.apache.org/jira/browse/HIVE-28447 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Priority: Major > > The tableExists in HiveMetaStoreClient is to check whether a table exists, in > current implementation an authorization filtering is performed on both the > client and server side, this API would return false even though there is such > a table in HMS due to this filter, which is against the role of this method. > As this method doesn't breach any internal details of the table, so it would > be better to remove the filtering, and make it accept the pattern based > tableName or databaseName. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-28371) Optimize add partitions authorization in HiveMetaStore
[ https://issues.apache.org/jira/browse/HIVE-28371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-28371. -- Fix Version/s: 4.1.0 Resolution: Fixed > Optimize add partitions authorization in HiveMetaStore > -- > > Key: HIVE-28371 > URL: https://issues.apache.org/jira/browse/HIVE-28371 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > > Currently add_partitions() api sends all the partitions (new partitions and > existing partitions) that need to be added for authorization, instead, we can > optimize this by sending only the new partitions for authorization. > Impact: Alter table recover partitions collects all the available partitions > and sends it to Metastore to check if any new partitions can be added. If all > the partitions are sent for authorization irrespective of whether it exists > or not, the Authorization service will unnecessarily spend time on > authorizing already existing partitions. This can be avoided by only > authorizing new partitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28371) Optimize add partitions authorization in HiveMetaStore
Sai Hemanth Gantasala created HIVE-28371: Summary: Optimize add partitions authorization in HiveMetaStore Key: HIVE-28371 URL: https://issues.apache.org/jira/browse/HIVE-28371 Project: Hive Issue Type: Improvement Components: Standalone Metastore Reporter: Sai Hemanth Gantasala Assignee: Sai Hemanth Gantasala Currently add_partitions() api sends all the partitions (new partitions and existing partitions) that need to be added for authorization, instead, we can optimize this by sending only the new partitions for authorization. Impact: Alter table recover partitions collects all the available partitions and sends it to Metastore to check if any new partitions can be added. If all the partitions are sent for authorization irrespective of whether it exists or not, the Authorization service will unnecessarily spend time on authorizing already existing partitions. This can be avoided by only authorizing new partitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-28349) SHOW TABLES with invalid connector, giving 0 results, instead of failing
[
https://issues.apache.org/jira/browse/HIVE-28349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-28349 started by Sai Hemanth Gantasala.
> SHOW TABLES with invalid connector, giving 0 results, instead of failing
>
>
> Key: HIVE-28349
> URL: https://issues.apache.org/jira/browse/HIVE-28349
> Project: Hive
> Issue Type: Sub-task
> Components: Hive, Standalone Metastore
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Fix For: 4.0.0
>
>
> SHOW TABLES with invalid connector, giving 0 results, instead of failing
> Steps to repro:
> {code:java}
> drop connector postgres_connector;
> create connector postgres_connector type 'postgres' url
> 'jdbc:postgresql://1.1.1.1:31462' with DCPROPERTIES
> ("hive.sql.dbcp.username"="root", "hive.sql.dbcp.password"="cloudera");
> drop database pg_hive_testing;
> create remote database pg_hive_testing using postgres_connector with
> DBPROPERTIES ("connector.remoteDbName"="postgres");
> show tables in pg_hive_testing; {code}
> The last query gives 0 rows (not a failure).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work started] (HIVE-28350) Drop remote database succeeds but fails while deleting data under
[
https://issues.apache.org/jira/browse/HIVE-28350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-28350 started by Sai Hemanth Gantasala.
> Drop remote database succeeds but fails while deleting data under
> -
>
> Key: HIVE-28350
> URL: https://issues.apache.org/jira/browse/HIVE-28350
> Project: Hive
> Issue Type: Sub-task
> Components: Hive, Standalone Metastore
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
>
> Drop remote database operation succeeds but fails towards the end while
> clearing data under the database's location because while fetching database
> object via JDO we don't seem to set the 'locationUri' field.
> {code:java}
> > drop database pg_hive_tests;
> INFO : Compiling
> command(queryId=hive_20240625161645_bbe11908-8d1c-46d7-9a02-1ef2091e1b86):
> drop database pg_hive_tests
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240625161645_bbe11908-8d1c-46d7-9a02-1ef2091e1b86);
> Time taken: 0.115 seconds
> INFO : Executing
> command(queryId=hive_20240625161645_bbe11908-8d1c-46d7-9a02-1ef2091e1b86):
> drop database pg_hive_tests
> INFO : Starting task [Stage-0:DDL] in serial mode
> ERROR : Failed
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:java.lang.IllegalArgumentException: Can not create a
> Path from a null string)
> at org.apache.hadoop.hive.ql.metadata.Hive.dropDatabase(Hive.java:716)
> ~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at
> org.apache.hadoop.hive.ql.ddl.database.drop.DropDatabaseOperation.execute(DropDatabaseOperation.java:51)
> ~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84)
> ~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> ~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> ~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:356)
> ~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:329)
> ~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> ~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:107)
> ~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:813)
> ~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:550)
> ~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:544)
> ~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:190)
> ~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:235)
> ~[hive-service-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:92)
> ~[hive-service-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:340)
> ~[hive-service-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_232]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_232]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> ~[hadoop-common-3.1.1.7.2.18.0-641.jar:?]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:360)
> ~[hive-service-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_232]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_232]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:
[jira] [Created] (HIVE-28350) Drop remote database succeeds but fails while deleting data under
Sai Hemanth Gantasala created HIVE-28350:
Summary: Drop remote database succeeds but fails while deleting
data under
Key: HIVE-28350
URL: https://issues.apache.org/jira/browse/HIVE-28350
Project: Hive
Issue Type: Sub-task
Components: Hive, Standalone Metastore
Reporter: Sai Hemanth Gantasala
Assignee: Sai Hemanth Gantasala
Drop remote database operation succeeds but fails towards the end while
clearing data under the database's location because while fetching database
object via JDO we don't seem to set the 'locationUri' field.
{code:java}
> drop database pg_hive_tests;
INFO : Compiling
command(queryId=hive_20240625161645_bbe11908-8d1c-46d7-9a02-1ef2091e1b86): drop
database pg_hive_tests
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling
command(queryId=hive_20240625161645_bbe11908-8d1c-46d7-9a02-1ef2091e1b86); Time
taken: 0.115 seconds
INFO : Executing
command(queryId=hive_20240625161645_bbe11908-8d1c-46d7-9a02-1ef2091e1b86): drop
database pg_hive_tests
INFO : Starting task [Stage-0:DDL] in serial mode
ERROR : Failed
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:java.lang.IllegalArgumentException: Can not create a Path
from a null string)
at org.apache.hadoop.hive.ql.metadata.Hive.dropDatabase(Hive.java:716)
~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at
org.apache.hadoop.hive.ql.ddl.database.drop.DropDatabaseOperation.execute(DropDatabaseOperation.java:51)
~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84)
~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:356)
~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:329)
~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:107)
~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:813)
~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:550)
~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:544)
~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:190)
~[hive-exec-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:235)
~[hive-service-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at
org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:92)
~[hive-service-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:340)
~[hive-service-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_232]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_232]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
~[hadoop-common-3.1.1.7.2.18.0-641.jar:?]
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:360)
~[hive-service-3.1.3000.7.2.18.0-641.jar:3.1.3000.7.2.18.0-641]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_232]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_232]
Caused by: org.apache.hadoop.hive.metastore.api.MetaException:
java.lang.IllegalArgumentException: Can not create a Path from a null string
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$drop_database
[jira] [Created] (HIVE-28349) SHOW TABLES with invalid connector, giving 0 results, instead of failing
Sai Hemanth Gantasala created HIVE-28349:
Summary: SHOW TABLES with invalid connector, giving 0 results,
instead of failing
Key: HIVE-28349
URL: https://issues.apache.org/jira/browse/HIVE-28349
Project: Hive
Issue Type: Sub-task
Components: Hive, Standalone Metastore
Reporter: Sai Hemanth Gantasala
Assignee: Sai Hemanth Gantasala
Fix For: 4.0.0
SHOW TABLES with invalid connector, giving 0 results, instead of failing
Steps to repro:
{code:java}
drop connector postgres_connector;
create connector postgres_connector type 'postgres' url
'jdbc:postgresql://1.1.1.1:31462' with DCPROPERTIES
("hive.sql.dbcp.username"="root", "hive.sql.dbcp.password"="cloudera");
drop database pg_hive_testing;
create remote database pg_hive_testing using postgres_connector with
DBPROPERTIES ("connector.remoteDbName"="postgres");
show tables in pg_hive_testing; {code}
The last query gives 0 rows (not a failure).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-28325) Lack of "owner" in HivePrivilegeObject causes Ranger slowness at compilation time
[ https://issues.apache.org/jira/browse/HIVE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855154#comment-17855154 ] Sai Hemanth Gantasala commented on HIVE-28325: -- [~scarlin] - Thanks for your contribution. The patch has been merged into the master branch. Thanks. > Lack of "owner" in HivePrivilegeObject causes Ranger slowness at compilation > time > - > > Key: HIVE-28325 > URL: https://issues.apache.org/jira/browse/HIVE-28325 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Steve Carlin >Assignee: Steve Carlin >Priority: Major > Labels: pull-request-available > > There is a HivePrivilegeObject created in the SemanticAnalyzer that is used > for Ranger calls. > Ranger uses the owner as a filter when searching for objects. When the owner > is not passed in, Ranger calls get slowed down noticeably, causing a slowdown > in compilation time. > > This is related to HIVE-27285 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-28325) Lack of "owner" in HivePrivilegeObject causes Ranger slowness at compilation time
[ https://issues.apache.org/jira/browse/HIVE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-28325. -- Fix Version/s: 4.0.1 Resolution: Fixed > Lack of "owner" in HivePrivilegeObject causes Ranger slowness at compilation > time > - > > Key: HIVE-28325 > URL: https://issues.apache.org/jira/browse/HIVE-28325 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Steve Carlin >Assignee: Steve Carlin >Priority: Major > Labels: pull-request-available > Fix For: 4.0.1 > > > There is a HivePrivilegeObject created in the SemanticAnalyzer that is used > for Ranger calls. > Ranger uses the owner as a filter when searching for objects. When the owner > is not passed in, Ranger calls get slowed down noticeably, causing a slowdown > in compilation time. > > This is related to HIVE-27285 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-28325) Lack of "owner" in HivePrivilegeObject causes Ranger slowness at compilation time
[ https://issues.apache.org/jira/browse/HIVE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala reassigned HIVE-28325: Assignee: Steve Carlin > Lack of "owner" in HivePrivilegeObject causes Ranger slowness at compilation > time > - > > Key: HIVE-28325 > URL: https://issues.apache.org/jira/browse/HIVE-28325 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Steve Carlin >Assignee: Steve Carlin >Priority: Major > Labels: pull-request-available > > There is a HivePrivilegeObject created in the SemanticAnalyzer that is used > for Ranger calls. > Ranger uses the owner as a filter when searching for objects. When the owner > is not passed in, Ranger calls get slowed down noticeably, causing a slowdown > in compilation time. > > This is related to HIVE-27285 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28294) drop database cascade operation can skip client side filtering while fetching tables in db
[ https://issues.apache.org/jira/browse/HIVE-28294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala updated HIVE-28294: - Description: Drop database cascade operation fetches all tables in the DB, while doing so we perform client-side filtering on the tables. We can avoid client-side filtering as we anyway authorize on the tables in the DB for the drop operation. Also, we need to add functions in the database for authorization before dropping the database. was:Drop database cascade operation fetches all tables in the DB, while doing so we perform client-side filtering on the tables. We can avoid client-side filtering as we anyway authorize on the tables in the DB for the drop operation. > drop database cascade operation can skip client side filtering while fetching > tables in db > -- > > Key: HIVE-28294 > URL: https://issues.apache.org/jira/browse/HIVE-28294 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > > Drop database cascade operation fetches all tables in the DB, while doing so > we perform client-side filtering on the tables. We can avoid client-side > filtering as we anyway authorize on the tables in the DB for the drop > operation. > Also, we need to add functions in the database for authorization before > dropping the database. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28294) drop database cascade operation can skip client side filtering while fetching tables in db
Sai Hemanth Gantasala created HIVE-28294: Summary: drop database cascade operation can skip client side filtering while fetching tables in db Key: HIVE-28294 URL: https://issues.apache.org/jira/browse/HIVE-28294 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Sai Hemanth Gantasala Assignee: Sai Hemanth Gantasala Drop database cascade operation fetches all tables in the DB, while doing so we perform client-side filtering on the tables. We can avoid client-side filtering as we anyway authorize on the tables in the DB for the drop operation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28137) Append Multi-Level Partition Filtering in Direct SQL Implementation of getPartitionsByFilter
[ https://issues.apache.org/jira/browse/HIVE-28137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17831559#comment-17831559 ] Sai Hemanth Gantasala commented on HIVE-28137: -- [~dzcxzl] - Can you add some description to the Jira? > Append Multi-Level Partition Filtering in Direct SQL Implementation of > getPartitionsByFilter > > > Key: HIVE-28137 > URL: https://issues.apache.org/jira/browse/HIVE-28137 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: dzcxzl >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27663) implement array_position UDF in Hive
[
https://issues.apache.org/jira/browse/HIVE-27663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved HIVE-27663.
--
Fix Version/s: 4.1.0
Resolution: Fixed
> implement array_position UDF in Hive
>
>
> Key: HIVE-27663
> URL: https://issues.apache.org/jira/browse/HIVE-27663
> Project: Hive
> Issue Type: Sub-task
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> array_position(array, element) - Returns the position of the first occurrence
> of element in array.
> Array indexing starts at 1. If the element value is NULL, a NULL is returned.
>
> Example:
> {noformat}
> > SELECT array_position(array(1, 2, 3,4,2), 2) FROM src;
> 2{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27663) implement array_position UDF in Hive
[
https://issues.apache.org/jira/browse/HIVE-27663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824213#comment-17824213
]
Sai Hemanth Gantasala commented on HIVE-27663:
--
[~tarak271] - Thanks for the contribution. The patch has been merged into the
master branch.
> implement array_position UDF in Hive
>
>
> Key: HIVE-27663
> URL: https://issues.apache.org/jira/browse/HIVE-27663
> Project: Hive
> Issue Type: Sub-task
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> array_position(array, element) - Returns the position of the first occurrence
> of element in array.
> Array indexing starts at 1. If the element value is NULL, a NULL is returned.
>
> Example:
> {noformat}
> > SELECT array_position(array(1, 2, 3,4,2), 2) FROM src;
> 2{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-28074) Normalize db/table name in the alter table operation in HMS
[
https://issues.apache.org/jira/browse/HIVE-28074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala reassigned HIVE-28074:
Assignee: Sai Hemanth Gantasala
> Normalize db/table name in the alter table operation in HMS
> ---
>
> Key: HIVE-28074
> URL: https://issues.apache.org/jira/browse/HIVE-28074
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
>
> Normalize db/table names in alter table event in the HMS.
> In create_table_core(), we do the following
> {code:java}
> tbl.setDbName(normalizeIdentifier(tbl.getDbName()));
> tbl.setTableName(normalizeIdentifier(tbl.getTableName())); {code}
> But in the alter_table, we are just taking case insensititive names and
> firing the alter_table event
> {code:java}
> firePreEvent(new PreAlterTableEvent(oldt, newTable, this)); {code}
> As a result, event consumers of HMS like Impala would face issues with the
> case insensitive names while consuming events.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-28074) Normalize db/table name in the alter table operation in HMS
[
https://issues.apache.org/jira/browse/HIVE-28074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17816814#comment-17816814
]
Sai Hemanth Gantasala commented on HIVE-28074:
--
cc [~dengzh] [~ngangam]
> Normalize db/table name in the alter table operation in HMS
> ---
>
> Key: HIVE-28074
> URL: https://issues.apache.org/jira/browse/HIVE-28074
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Sai Hemanth Gantasala
>Priority: Major
>
> Normalize db/table names in alter table event in the HMS.
> In create_table_core(), we do the following
> {code:java}
> tbl.setDbName(normalizeIdentifier(tbl.getDbName()));
> tbl.setTableName(normalizeIdentifier(tbl.getTableName())); {code}
> But in the alter_table, we are just taking case insensititive names and
> firing the alter_table event
> {code:java}
> firePreEvent(new PreAlterTableEvent(oldt, newTable, this)); {code}
> As a result, event consumers of HMS like Impala would face issues with the
> case insensitive names while consuming events.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28074) Normalize db/table name in the alter table operation in HMS
Sai Hemanth Gantasala created HIVE-28074:
Summary: Normalize db/table name in the alter table operation in
HMS
Key: HIVE-28074
URL: https://issues.apache.org/jira/browse/HIVE-28074
Project: Hive
Issue Type: Improvement
Components: Standalone Metastore
Reporter: Sai Hemanth Gantasala
Normalize db/table names in alter table event in the HMS.
In create_table_core(), we do the following
{code:java}
tbl.setDbName(normalizeIdentifier(tbl.getDbName()));
tbl.setTableName(normalizeIdentifier(tbl.getTableName())); {code}
But in the alter_table, we are just taking case insensititive names and firing
the alter_table event
{code:java}
firePreEvent(new PreAlterTableEvent(oldt, newTable, this)); {code}
As a result, event consumers of HMS like Impala would face issues with the case
insensitive names while consuming events.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28060) Support optional argument for dropping stats in the truncate_table() of HMS
Sai Hemanth Gantasala created HIVE-28060: Summary: Support optional argument for dropping stats in the truncate_table() of HMS Key: HIVE-28060 URL: https://issues.apache.org/jira/browse/HIVE-28060 Project: Hive Issue Type: Wish Components: Standalone Metastore Reporter: Sai Hemanth Gantasala Assignee: Sai Hemanth Gantasala Certain clients like Impala rely on truncate_table() API in HMS to take care of truncate operations for replicated tables in Impala. There is an option 'set DELETE_STATS_IN_TRUNCATE=0;' to specify not to delete stats during the truncate operation. It would be good to support this optional argument whether to delete stats or not during the truncate table operation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26837) CTLT with hive.create.as.external.legacy as true creates managed table instead of external table
[ https://issues.apache.org/jira/browse/HIVE-26837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala updated HIVE-26837: - Fix Version/s: 4.0.0 > CTLT with hive.create.as.external.legacy as true creates managed table > instead of external table > > > Key: HIVE-26837 > URL: https://issues.apache.org/jira/browse/HIVE-26837 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > When CTLT is used with the config hive.create.as.external.legacy=true, it > still creates managed table by default. Use below to reproduce. > create external table test_ext(empno int, name string) partitioned by(dept > string) stored as orc; > desc formatted test_ext; > set hive.create.as.external.legacy=true; > create table test_external like test_ext; > desc formatted test_external; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27827) Improve performance of direct SQL implement for getPartitionsByFilter
[ https://issues.apache.org/jira/browse/HIVE-27827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808758#comment-17808758 ] Sai Hemanth Gantasala commented on HIVE-27827: -- [~wechar] - The patch has been merged into the master branch. Thanks for your contribution. > Improve performance of direct SQL implement for getPartitionsByFilter > - > > Key: HIVE-27827 > URL: https://issues.apache.org/jira/browse/HIVE-27827 > Project: Hive > Issue Type: Improvement >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27827) Improve performance of direct SQL implement for getPartitionsByFilter
[ https://issues.apache.org/jira/browse/HIVE-27827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala updated HIVE-27827: - Fix Version/s: 4.1.0 > Improve performance of direct SQL implement for getPartitionsByFilter > - > > Key: HIVE-27827 > URL: https://issues.apache.org/jira/browse/HIVE-27827 > Project: Hive > Issue Type: Improvement >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27827) Improve performance of direct SQL implement for getPartitionsByFilter
[ https://issues.apache.org/jira/browse/HIVE-27827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-27827. -- Resolution: Fixed > Improve performance of direct SQL implement for getPartitionsByFilter > - > > Key: HIVE-27827 > URL: https://issues.apache.org/jira/browse/HIVE-27827 > Project: Hive > Issue Type: Improvement >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28001) Fix the flaky test TestLeaderElection
[ https://issues.apache.org/jira/browse/HIVE-28001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807876#comment-17807876 ] Sai Hemanth Gantasala commented on HIVE-28001: -- [~dengzh] - I think it would be good to post the actual failure of the test for future reference in the Jira details. The links posted in the Jira can expire at any time. > Fix the flaky test TestLeaderElection > - > > Key: HIVE-28001 > URL: https://issues.apache.org/jira/browse/HIVE-28001 > Project: Hive > Issue Type: Test >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > > The TestLeaderElection is failing sometimes, example: > [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2032/tests] > [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4995/3/tests] > [http://ci.hive.apache.org/job/hive-flaky-check/796/] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26537) Deprecate older APIs in the HMS
[ https://issues.apache.org/jira/browse/HIVE-26537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799112#comment-17799112 ] Sai Hemanth Gantasala commented on HIVE-26537: -- [~dkuzmenko] - Thanks for the reminder. Revived the patch. Will get it reviewed again. > Deprecate older APIs in the HMS > --- > > Key: HIVE-26537 > URL: https://issues.apache.org/jira/browse/HIVE-26537 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0-alpha-1, 4.0.0-alpha-2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Critical > Labels: pull-request-available > Time Spent: 7h 10m > Remaining Estimate: 0h > > This Jira is to track the clean-up(deprecate older APIs and point the HMS > client to the newer APIs) work in the hive metastore server. > More details will be added here soon. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-25803) URL Mapping appends default Fs scheme even for LOCAL DIRECTORY ops
[
https://issues.apache.org/jira/browse/HIVE-25803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798377#comment-17798377
]
Sai Hemanth Gantasala commented on HIVE-25803:
--
[~ayushtkn] – Please feel free to assign this to you to work on this.
> URL Mapping appends default Fs scheme even for LOCAL DIRECTORY ops
> --
>
> Key: HIVE-25803
> URL: https://issues.apache.org/jira/browse/HIVE-25803
> Project: Hive
> Issue Type: Bug
> Components: Authorization, HiveServer2
>Affects Versions: 4.0.0
>Reporter: Soumitra Sulav
>Assignee: Sai Hemanth Gantasala
>Priority: Critical
>
> Repro steps:
> Connect to beeline
> {code:java}
> beeline -u
> "jdbc:hive2://quasar-pxlypi-2.quasar-pxlypi.root.hwx.site:10001/;principal=hive/_h...@root.hwx.site;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;trustStorePassword=VOAnRk5l4oXsg0upJ1ApscSuNksirOKgyhJvoPv2o4j;transportMode=http;httpPath=cliservice;"
> {code}
>
> Create a test table and run insert on local
> {code:java}
> > create table dual (id int);
> > insert overwrite local directory "/tmp/" select * from dual;
> {code}
> {code:java}
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [hrt_qa] does not have [ALL] privilege on
> [hdfs://ns1/tmp] (state=42000,code=4)
> {code}
> It always appends hdfs:// to the path even if the operation is meant for
> local directory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27892) Hive "insert overwrite table" for multiple partition table issue
[ https://issues.apache.org/jira/browse/HIVE-27892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala updated HIVE-27892: - Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) [~mkunwar] - Thank you for your contribution. The patch has been merged to the master branch. > Hive "insert overwrite table" for multiple partition table issue > > > Key: HIVE-27892 > URL: https://issues.apache.org/jira/browse/HIVE-27892 > Project: Hive > Issue Type: Bug >Reporter: Mayank Kunwar >Assignee: Mayank Kunwar >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Authorization is not working for Hive "insert overwrite table" for multiple > partition table. > Steps to reproduce the issue: > 1) CREATE EXTERNAL TABLE Part (eid int, name int) > PARTITIONED BY (position int, dept int); > 2) SET hive.exec.dynamic.partition.mode=nonstrict; > 3) INSERT INTO TABLE PART PARTITION (position,DEPT) > SELECT 1,1,1,1; > 4) select * from part; > create a test user test123, and grant test123 only Select permission for db > default, table Part and column * . > 1) insert overwrite table part partition(position=2,DEPT=2) select 2,2; > This will failed as expected. > 2) insert overwrite table part partition(position,DEPT) select 2,2,2,2; > This will failed as expected. > 3) insert overwrite table part partition(position=2,DEPT) select 2,2,2; > But this will succeed and no audit in Ranger, which means no authorization > happened when this query was executed. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-25803) URL Mapping appends hdfs:// even for LOCAL DIRECTORY ops
[
https://issues.apache.org/jira/browse/HIVE-25803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794389#comment-17794389
]
Sai Hemanth Gantasala commented on HIVE-25803:
--
[~dkuzmenko] - Yeah, it is still a problem. This needs to be addressed.
> URL Mapping appends hdfs:// even for LOCAL DIRECTORY ops
>
>
> Key: HIVE-25803
> URL: https://issues.apache.org/jira/browse/HIVE-25803
> Project: Hive
> Issue Type: Bug
> Components: Authorization, HiveServer2
>Affects Versions: 4.0.0
>Reporter: Soumitra Sulav
>Assignee: Sai Hemanth Gantasala
>Priority: Major
>
> Repro steps:
> Connect to beeline
> {code:java}
> beeline -u
> "jdbc:hive2://quasar-pxlypi-2.quasar-pxlypi.root.hwx.site:10001/;principal=hive/_h...@root.hwx.site;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;trustStorePassword=VOAnRk5l4oXsg0upJ1ApscSuNksirOKgyhJvoPv2o4j;transportMode=http;httpPath=cliservice;"
> {code}
>
> Create a test table and run insert on local
> {code:java}
> > create table dual (id int);
> > insert overwrite local directory "/tmp/" select * from dual;
> {code}
> {code:java}
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [hrt_qa] does not have [ALL] privilege on
> [hdfs://ns1/tmp] (state=42000,code=4)
> {code}
> It always appends hdfs:// to the path even if the operation is meant for
> local directory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27499) Add optional table name filter to the HMS notification fetch API
[
https://issues.apache.org/jira/browse/HIVE-27499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved HIVE-27499.
--
Fix Version/s: 4.0.0-beta-1
Resolution: Fixed
> Add optional table name filter to the HMS notification fetch API
>
>
> Key: HIVE-27499
> URL: https://issues.apache.org/jira/browse/HIVE-27499
> Project: Hive
> Issue Type: New Feature
>Reporter: Quanlong Huang
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-beta-1
>
>
> The current notification fetch API does not support filters on table name:
> {code:java}
> struct NotificationEventRequest {
> 1: required i64 lastEvent,
> 2: optional i32 maxEvents,
> 3: optional list eventTypeSkipList,
> }
> NotificationEventResponse get_next_notification(1:NotificationEventRequest
> rqst)
> {code}
> Engines that cache HMS metadata and file metadata might want to know whether
> they have synced to the latest events on some given tables.
> In order to efficiently fetch events specific to a table, we should enhance
> the notification fetch API to include an optional table name in the request
> so that only events for that given table are returned. The NOTIFICATION_LOG
> table in the HMS should also have an index on table_name so that API calls
> with the filter do not do a table scan.
> With the index added, we can also extend the NotificationEventsCountRequest
> with an optional table name, and extend the response to carry the largest
> event id on that table. Currently they are:
> {code:java}
> struct NotificationEventsCountRequest {
> 1: required i64 fromEventId,
> 2: required string dbName,
> 3: optional string catName,
> 4: optional i64 toEventId,
> 5: optional i64 limit
> }
> struct NotificationEventsCountResponse {
> 1: required i64 eventsCount,
> }
> NotificationEventsCountResponse
> get_notification_events_count(1:NotificationEventsCountRequest rqst)
> {code}
> Similar API for multiple tables (given a table lists) will also be helpful
> since a query usually involves several tables.
> CC [~hemanth619], [~ngangam]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27499) Add optional table name filter to the HMS notification fetch API
[
https://issues.apache.org/jira/browse/HIVE-27499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782221#comment-17782221
]
Sai Hemanth Gantasala commented on HIVE-27499:
--
The patch has been merged into the master branch. Thanks for the review
[~ngangam], [~dengzh]
> Add optional table name filter to the HMS notification fetch API
>
>
> Key: HIVE-27499
> URL: https://issues.apache.org/jira/browse/HIVE-27499
> Project: Hive
> Issue Type: New Feature
>Reporter: Quanlong Huang
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
>
> The current notification fetch API does not support filters on table name:
> {code:java}
> struct NotificationEventRequest {
> 1: required i64 lastEvent,
> 2: optional i32 maxEvents,
> 3: optional list eventTypeSkipList,
> }
> NotificationEventResponse get_next_notification(1:NotificationEventRequest
> rqst)
> {code}
> Engines that cache HMS metadata and file metadata might want to know whether
> they have synced to the latest events on some given tables.
> In order to efficiently fetch events specific to a table, we should enhance
> the notification fetch API to include an optional table name in the request
> so that only events for that given table are returned. The NOTIFICATION_LOG
> table in the HMS should also have an index on table_name so that API calls
> with the filter do not do a table scan.
> With the index added, we can also extend the NotificationEventsCountRequest
> with an optional table name, and extend the response to carry the largest
> event id on that table. Currently they are:
> {code:java}
> struct NotificationEventsCountRequest {
> 1: required i64 fromEventId,
> 2: required string dbName,
> 3: optional string catName,
> 4: optional i64 toEventId,
> 5: optional i64 limit
> }
> struct NotificationEventsCountResponse {
> 1: required i64 eventsCount,
> }
> NotificationEventsCountResponse
> get_notification_events_count(1:NotificationEventsCountRequest rqst)
> {code}
> Similar API for multiple tables (given a table lists) will also be helpful
> since a query usually involves several tables.
> CC [~hemanth619], [~ngangam]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27189) Remove duplicate debug log in Hive.isSubDIr
[
https://issues.apache.org/jira/browse/HIVE-27189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala updated HIVE-27189:
-
Fix Version/s: 4.0.0-beta-1
(was: All Versions)
> Remove duplicate debug log in Hive.isSubDIr
> ---
>
> Key: HIVE-27189
> URL: https://issues.apache.org/jira/browse/HIVE-27189
> Project: Hive
> Issue Type: Improvement
>Reporter: shuyouZZ
>Assignee: shuyouZZ
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-beta-1
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> In class {{{}org.apache.hadoop.hive.ql.metadata.HIve{}}}, invoke method
> {{isSubDir}} will print twice
> {code:java}
> LOG.debug("The source path is " + fullF1 + " and the destination path is " +
> fullF2);{code}
> we should remove the duplicate debug log.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27676) Reuse the add_partitions logic for add_partition in ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-27676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved HIVE-27676.
--
Resolution: Fixed
[~wechar] - Patch merged to the master branch. Thanks for the contribution.
> Reuse the add_partitions logic for add_partition in ObjectStore
> ---
>
> Key: HIVE-27676
> URL: https://issues.apache.org/jira/browse/HIVE-27676
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0-beta-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
>
> HIVE-26035 implements direct SQL for {{add_partitions}} to improve
> performance, we can also reuse this logic for {{{}add_partition{}}} with
> following benefits:
> * Get the performance improvement in direct SQL
> * Code cleaner, reduce the duplicate code.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27525) Ease the write permissions on external table during create table operation
[ https://issues.apache.org/jira/browse/HIVE-27525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-27525. -- Fix Version/s: 4.0.0 Resolution: Fixed The patch has been merged into the master branch. Thanks for the contribution [~rtrivedi12] . > Ease the write permissions on external table during create table operation > -- > > Key: HIVE-27525 > URL: https://issues.apache.org/jira/browse/HIVE-27525 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Riju Trivedi >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > During the creation of external tables with a specified location, the general > expectation is that the data is already present or the data might be > externally added to the location without involving HMS. So, it is really not > required to have read and write permissions on an external table during the > creation time. > This enhancement can address security concerns where currently the users had > to be granted unnecessary write permissions on an external file location when > the table is only used for reading the data. > Update/delete operations would anyway require write permissions. > CTAS query with location specified is expected to fail at runtime (trying to > create staging directory under table location) without WRITE permissions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26537) Deprecate older APIs in the HMS
[ https://issues.apache.org/jira/browse/HIVE-26537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17760153#comment-17760153 ] Sai Hemanth Gantasala commented on HIVE-26537: -- [~aturoczy] - Sorry for the delay. I certainly don't want to leave this work behind. I would like to put [HIVE-26893|https://github.com/apache/hive/pull/4638#top] before this patch goes in. > Deprecate older APIs in the HMS > --- > > Key: HIVE-26537 > URL: https://issues.apache.org/jira/browse/HIVE-26537 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0-alpha-1, 4.0.0-alpha-2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Critical > Labels: hive-4.0.0-must, pull-request-available > Time Spent: 7h 10m > Remaining Estimate: 0h > > This Jira is to track the clean-up(deprecate older APIs and point the HMS > client to the newer APIs) work in the hive metastore server. > More details will be added here soon. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27628) implement array_remove UDF in Hive
[
https://issues.apache.org/jira/browse/HIVE-27628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17759980#comment-17759980
]
Sai Hemanth Gantasala commented on HIVE-27628:
--
The patch has been merged into the master branch. Thanks for the contribution
[~tarak271].
> implement array_remove UDF in Hive
> --
>
> Key: HIVE-27628
> URL: https://issues.apache.org/jira/browse/HIVE-27628
> Project: Hive
> Issue Type: Sub-task
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
>
> *array_remove(array, element)*
> Removes all occurrences of {{element}} from {{{}array{}}}.
>
> Example:
> {noformat}
> > SELECT array_remove(array(1, 2, 3,4,2), 2) FROM src;
> [1,3,4]{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-22961) Drop function in Hive should not send request for drop database to Ranger plugin.
[
https://issues.apache.org/jira/browse/HIVE-22961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved HIVE-22961.
--
Fix Version/s: 4.0.0
Resolution: Fixed
> Drop function in Hive should not send request for drop database to Ranger
> plugin.
> --
>
> Key: HIVE-22961
> URL: https://issues.apache.org/jira/browse/HIVE-22961
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Issue here is how HIVE sends the "HivePrivilegeObjects" to Ranger when DROP
> fUNTION is done. This is different from how DROP TABLE is done.
> DROP TABLE the following is the request:
> {code:java}
> 'checkPrivileges':{'hiveOpType':DROPTABLE,
> 'inputHObjs':['HivePrivilegeObject':{'type':TABLE_OR_VIEW, 'dbName':testdemo,
> 'objectType':TABLE_OR_VIEW, 'objectName':t1, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER, 'owner':systest}],
> 'outputHObjs':['HivePrivilegeObject':{'type':TABLE_OR_VIEW,
> 'dbName':testdemo, 'objectType':TABLE_OR_VIEW, 'objectName':t1, 'columns':[],
> 'partKeys':[], 'commandParams':[], 'actionType':OTHER, 'owner':systest}],
> 'context':{'clientType':HIVESERVER2, 'commandString':drop table t1,
> 'ipAddress':10.65.42.125, 'forwardedAddresses':null,
> 'sessionString':58f89a16-2df5-4124-af0e-913aabbefe06}, 'user':systest,
> 'groups':[systest, wheel]}{code}
> Where as in DROP FUNCTION:
> {code:java}
> {'hiveOpType':DROPFUNCTION,
> 'inputHObjs':['HivePrivilegeObject':{'type':FUNCTION, 'dbName':udfdemo,
> 'objectType':FUNCTION, 'objectName':aes1, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER, 'owner':null}],
> 'outputHObjs':['HivePrivilegeObject':{'type':DATABASE, 'dbName':udfdemo,
> 'objectType':DATABASE, 'objectName':null, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER,
> 'owner':systest},'HivePrivilegeObject':{'type':FUNCTION, 'dbName':udfdemo,
> 'objectType':FUNCTION, 'objectName':aes1, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER, 'owner':null}],
> 'context':{'clientType':HIVESERVER2, 'commandString':drop function
> udfdemo.aes1, 'ipAddress':10.65.42.125, 'forwardedAddresses':null,
> 'sessionString':442ca4d3-f34a-470c-878a-18542b99016c}, 'user':systest,
> 'groups':[systest, wheel]}
> {code}
> in DROP function in outputHObjs, there is this addition, DATABASE object
> which should not there and this causes the Ranger requested to be generated
> differently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-22961) Drop function in Hive should not send request for drop database to Ranger plugin.
[
https://issues.apache.org/jira/browse/HIVE-22961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17759267#comment-17759267
]
Sai Hemanth Gantasala commented on HIVE-22961:
--
[~rtrivedi12] - Patch merged to master branch. Thanks for contribution.
> Drop function in Hive should not send request for drop database to Ranger
> plugin.
> --
>
> Key: HIVE-22961
> URL: https://issues.apache.org/jira/browse/HIVE-22961
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Issue here is how HIVE sends the "HivePrivilegeObjects" to Ranger when DROP
> fUNTION is done. This is different from how DROP TABLE is done.
> DROP TABLE the following is the request:
> {code:java}
> 'checkPrivileges':{'hiveOpType':DROPTABLE,
> 'inputHObjs':['HivePrivilegeObject':{'type':TABLE_OR_VIEW, 'dbName':testdemo,
> 'objectType':TABLE_OR_VIEW, 'objectName':t1, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER, 'owner':systest}],
> 'outputHObjs':['HivePrivilegeObject':{'type':TABLE_OR_VIEW,
> 'dbName':testdemo, 'objectType':TABLE_OR_VIEW, 'objectName':t1, 'columns':[],
> 'partKeys':[], 'commandParams':[], 'actionType':OTHER, 'owner':systest}],
> 'context':{'clientType':HIVESERVER2, 'commandString':drop table t1,
> 'ipAddress':10.65.42.125, 'forwardedAddresses':null,
> 'sessionString':58f89a16-2df5-4124-af0e-913aabbefe06}, 'user':systest,
> 'groups':[systest, wheel]}{code}
> Where as in DROP FUNCTION:
> {code:java}
> {'hiveOpType':DROPFUNCTION,
> 'inputHObjs':['HivePrivilegeObject':{'type':FUNCTION, 'dbName':udfdemo,
> 'objectType':FUNCTION, 'objectName':aes1, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER, 'owner':null}],
> 'outputHObjs':['HivePrivilegeObject':{'type':DATABASE, 'dbName':udfdemo,
> 'objectType':DATABASE, 'objectName':null, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER,
> 'owner':systest},'HivePrivilegeObject':{'type':FUNCTION, 'dbName':udfdemo,
> 'objectType':FUNCTION, 'objectName':aes1, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER, 'owner':null}],
> 'context':{'clientType':HIVESERVER2, 'commandString':drop function
> udfdemo.aes1, 'ipAddress':10.65.42.125, 'forwardedAddresses':null,
> 'sessionString':442ca4d3-f34a-470c-878a-18542b99016c}, 'user':systest,
> 'groups':[systest, wheel]}
> {code}
> in DROP function in outputHObjs, there is this addition, DATABASE object
> which should not there and this causes the Ranger requested to be generated
> differently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27523) Implement array_union UDF in Hive
[
https://issues.apache.org/jira/browse/HIVE-27523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751802#comment-17751802
]
Sai Hemanth Gantasala commented on HIVE-27523:
--
[~tarak271] - Thanks for the contribution. The patch is merged into the master
branch.
> Implement array_union UDF in Hive
> -
>
> Key: HIVE-27523
> URL: https://issues.apache.org/jira/browse/HIVE-27523
> Project: Hive
> Issue Type: Sub-task
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> *array_union(array1, array2)*
> Returns an array of the elements in the union of {{array1}} and {{array2}}
> without duplicates.
>
> {noformat}
> SELECT array_union(array(1, 2, 2, 3), array(1, 3, 5));
> [1,2,3,5]
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27523) Implement array_union UDF in Hive
[
https://issues.apache.org/jira/browse/HIVE-27523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved HIVE-27523.
--
Fix Version/s: 4.0.0
Resolution: Fixed
> Implement array_union UDF in Hive
> -
>
> Key: HIVE-27523
> URL: https://issues.apache.org/jira/browse/HIVE-27523
> Project: Hive
> Issue Type: Sub-task
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> *array_union(array1, array2)*
> Returns an array of the elements in the union of {{array1}} and {{array2}}
> without duplicates.
>
> {noformat}
> SELECT array_union(array(1, 2, 2, 3), array(1, 3, 5));
> [1,2,3,5]
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27359) Backport HIVE-26850: ColumnInfo of the source table should not be null for the alter view as query to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-27359. -- Fix Version/s: 3.2.0 Resolution: Fixed > Backport HIVE-26850: ColumnInfo of the source table should not be null for > the alter view as query to branch-3 > -- > > Key: HIVE-27359 > URL: https://issues.apache.org/jira/browse/HIVE-27359 > Project: Hive > Issue Type: Sub-task >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Fix For: 3.2.0 > > > The ColumnInfo for the source table of the alter view as query is coming as > null and as a result, the column info is not present for the table. The > ColumnInfo should be non-null. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27499) Add optional table name filter to the HMS notification fetch API
[
https://issues.apache.org/jira/browse/HIVE-27499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala reassigned HIVE-27499:
Assignee: Sai Hemanth Gantasala
> Add optional table name filter to the HMS notification fetch API
>
>
> Key: HIVE-27499
> URL: https://issues.apache.org/jira/browse/HIVE-27499
> Project: Hive
> Issue Type: New Feature
>Reporter: Quanlong Huang
>Assignee: Sai Hemanth Gantasala
>Priority: Major
>
> The current notification fetch API does not support filters on table name:
> {code:java}
> struct NotificationEventRequest {
> 1: required i64 lastEvent,
> 2: optional i32 maxEvents,
> 3: optional list eventTypeSkipList,
> }
> NotificationEventResponse get_next_notification(1:NotificationEventRequest
> rqst)
> {code}
> Engines that cache HMS metadata and file metadata might want to know whether
> they have synced to the latest events on some given tables.
> In order to efficiently fetch events specific to a table, we should enhance
> the notification fetch API to include an optional table name in the request
> so that only events for that given table are returned. The NOTIFICATION_LOG
> table in the HMS should also have an index on table_name so that API calls
> with the filter do not do a table scan.
> With the index added, we can also extend the NotificationEventsCountRequest
> with an optional table name, and extend the response to carry the largest
> event id on that table. Currently they are:
> {code:java}
> struct NotificationEventsCountRequest {
> 1: required i64 fromEventId,
> 2: required string dbName,
> 3: optional string catName,
> 4: optional i64 toEventId,
> 5: optional i64 limit
> }
> struct NotificationEventsCountResponse {
> 1: required i64 eventsCount,
> }
> NotificationEventsCountResponse
> get_notification_events_count(1:NotificationEventsCountRequest rqst)
> {code}
> Similar API for multiple tables (given a table lists) will also be helpful
> since a query usually involves several tables.
> CC [~hemanth619], [~ngangam]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27504) Automate inclusion of 3-party licenses in binary distribution
[ https://issues.apache.org/jira/browse/HIVE-27504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-27504. -- Resolution: Fixed > Automate inclusion of 3-party licenses in binary distribution > - > > Key: HIVE-27504 > URL: https://issues.apache.org/jira/browse/HIVE-27504 > Project: Hive > Issue Type: Improvement >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > > Manually downloading and including licenses of bundled dependencies in the > binary distribution (as it is done currently under [binary-package-licenses > directory| > https://github.com/apache/hive/tree/bd02abc9eba3aa0cd3701b26d9169130551a12ef/binary-package-licenses]) > is error-prone and time consuming. > Moreover, dependencies and their versions change rapidly and it is hard to > keep the LICENSE documentation correct and up-to-date. At the moment, the > binary-package-licenses directory is stale; some bundled dependencies are > missing, others do not longer exist, and versions are not mentioned anywhere > so we cannot know if the license has changed. > Furthermore, every dependency change (add/remove/version bump) potentially > requires commits and modifications to the source code. > The goal of this ticket is to automate the download and inclusion of > 3rd-party licenses in the binary distribution to minimize/eliminate the need > for human intervention. As part of the automation the following issues should > be fixed as well: > 1. LICENSE/NOTICE should reflect exactly what lies inside each package > distribution (source & binary); ideally no more and definitely not less. > 2. LICENSES for bundled dependencies should be referenced from the main > LICENSE file not from NOTICE (or other places) as it is right now. > 3. For bundled dependencies their versions should be included cause different > version may use different licenses and it should be clear which version is > bundled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27504) Automate inclusion of 3-party licenses in binary distribution
[ https://issues.apache.org/jira/browse/HIVE-27504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748762#comment-17748762 ] Sai Hemanth Gantasala commented on HIVE-27504: -- Thanks for contributing an important patch for the release [~zabetak]. The patch has been merged into the master branch. > Automate inclusion of 3-party licenses in binary distribution > - > > Key: HIVE-27504 > URL: https://issues.apache.org/jira/browse/HIVE-27504 > Project: Hive > Issue Type: Improvement >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > > Manually downloading and including licenses of bundled dependencies in the > binary distribution (as it is done currently under [binary-package-licenses > directory| > https://github.com/apache/hive/tree/bd02abc9eba3aa0cd3701b26d9169130551a12ef/binary-package-licenses]) > is error-prone and time consuming. > Moreover, dependencies and their versions change rapidly and it is hard to > keep the LICENSE documentation correct and up-to-date. At the moment, the > binary-package-licenses directory is stale; some bundled dependencies are > missing, others do not longer exist, and versions are not mentioned anywhere > so we cannot know if the license has changed. > Furthermore, every dependency change (add/remove/version bump) potentially > requires commits and modifications to the source code. > The goal of this ticket is to automate the download and inclusion of > 3rd-party licenses in the binary distribution to minimize/eliminate the need > for human intervention. As part of the automation the following issues should > be fixed as well: > 1. LICENSE/NOTICE should reflect exactly what lies inside each package > distribution (source & binary); ideally no more and definitely not less. > 2. LICENSES for bundled dependencies should be referenced from the main > LICENSE file not from NOTICE (or other places) as it is right now. > 3. For bundled dependencies their versions should be included cause different > version may use different licenses and it should be clear which version is > bundled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27500) Normalize add_partitions related api in HMS
[
https://issues.apache.org/jira/browse/HIVE-27500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved HIVE-27500.
--
Fix Version/s: 4.0.0
Resolution: Fixed
> Normalize add_partitions related api in HMS
> ---
>
> Key: HIVE-27500
> URL: https://issues.apache.org/jira/browse/HIVE-27500
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {{HMSHandler}} has three APIs for add partitions: {{add_partitions_req}},
> {{add_partitions}} and {{add_partitions_pspec}}. Currently both
> {{add_partitions_req}} and {{add_partitions}} APIs invoke
> {{add_partitions_core}} function which was improved by direct sql in
> HIVE-26035.
> We can also invoke {{add_partitions_core}} function within
> {{add_partitions_pspec}} API for performance improvement.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27500) Normalize add_partitions related api in HMS
[
https://issues.apache.org/jira/browse/HIVE-27500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748271#comment-17748271
]
Sai Hemanth Gantasala commented on HIVE-27500:
--
[~wechar] - Thank you for the contribution. The patch has been merged into the
master branch.
> Normalize add_partitions related api in HMS
> ---
>
> Key: HIVE-27500
> URL: https://issues.apache.org/jira/browse/HIVE-27500
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
>
> {{HMSHandler}} has three APIs for add partitions: {{add_partitions_req}},
> {{add_partitions}} and {{add_partitions_pspec}}. Currently both
> {{add_partitions_req}} and {{add_partitions}} APIs invoke
> {{add_partitions_core}} function which was improved by direct sql in
> HIVE-26035.
> We can also invoke {{add_partitions_core}} function within
> {{add_partitions_pspec}} API for performance improvement.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-22961) Drop function in Hive should not send request for drop database to Ranger plugin.
[
https://issues.apache.org/jira/browse/HIVE-22961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala reassigned HIVE-22961:
Assignee: Sai Hemanth Gantasala (was: Sam An)
> Drop function in Hive should not send request for drop database to Ranger
> plugin.
> --
>
> Key: HIVE-22961
> URL: https://issues.apache.org/jira/browse/HIVE-22961
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Sai Hemanth Gantasala
>Priority: Major
>
> Issue here is how HIVE sends the "HivePrivilegeObjects" to Ranger when DROP
> fUNTION is done. This is different from how DROP TABLE is done.
> DROP TABLE the following is the request:
> {code:java}
> 'checkPrivileges':{'hiveOpType':DROPTABLE,
> 'inputHObjs':['HivePrivilegeObject':{'type':TABLE_OR_VIEW, 'dbName':testdemo,
> 'objectType':TABLE_OR_VIEW, 'objectName':t1, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER, 'owner':systest}],
> 'outputHObjs':['HivePrivilegeObject':{'type':TABLE_OR_VIEW,
> 'dbName':testdemo, 'objectType':TABLE_OR_VIEW, 'objectName':t1, 'columns':[],
> 'partKeys':[], 'commandParams':[], 'actionType':OTHER, 'owner':systest}],
> 'context':{'clientType':HIVESERVER2, 'commandString':drop table t1,
> 'ipAddress':10.65.42.125, 'forwardedAddresses':null,
> 'sessionString':58f89a16-2df5-4124-af0e-913aabbefe06}, 'user':systest,
> 'groups':[systest, wheel]}{code}
> Where as in DROP FUNCTION:
> {code:java}
> {'hiveOpType':DROPFUNCTION,
> 'inputHObjs':['HivePrivilegeObject':{'type':FUNCTION, 'dbName':udfdemo,
> 'objectType':FUNCTION, 'objectName':aes1, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER, 'owner':null}],
> 'outputHObjs':['HivePrivilegeObject':{'type':DATABASE, 'dbName':udfdemo,
> 'objectType':DATABASE, 'objectName':null, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER,
> 'owner':systest},'HivePrivilegeObject':{'type':FUNCTION, 'dbName':udfdemo,
> 'objectType':FUNCTION, 'objectName':aes1, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER, 'owner':null}],
> 'context':{'clientType':HIVESERVER2, 'commandString':drop function
> udfdemo.aes1, 'ipAddress':10.65.42.125, 'forwardedAddresses':null,
> 'sessionString':442ca4d3-f34a-470c-878a-18542b99016c}, 'user':systest,
> 'groups':[systest, wheel]}
> {code}
> in DROP function in outputHObjs, there is this addition, DATABASE object
> which should not there and this causes the Ranger requested to be generated
> differently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27525) Ease the write permissions on external table during create table operation
Sai Hemanth Gantasala created HIVE-27525: Summary: Ease the write permissions on external table during create table operation Key: HIVE-27525 URL: https://issues.apache.org/jira/browse/HIVE-27525 Project: Hive Issue Type: Improvement Components: Standalone Metastore Reporter: Sai Hemanth Gantasala Assignee: Sai Hemanth Gantasala During the creation of external tables with a specified location, the general expectation is that the data is already present or the data might be externally added to the location without involving HMS. So, it is really not required to have read and write permissions on an external table during the creation time. This enhancement can address security concerns where currently the users had to be grant unnecessary write permissions on an external file location when the table is only used for reading the data. Update/delete operations would anyway require write permissions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27482) Minor issues in array_intersect udf
[
https://issues.apache.org/jira/browse/HIVE-27482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17742487#comment-17742487
]
Sai Hemanth Gantasala commented on HIVE-27482:
--
Thank you for the contribution [~tarak271]. The patch has been merged to master
branch.
> Minor issues in array_intersect udf
> ---
>
> Key: HIVE-27482
> URL: https://issues.apache.org/jira/browse/HIVE-27482
> Project: Hive
> Issue Type: Sub-task
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Minor
> Labels: pull-request-available
>
> 1. SELECT array(1,2,3),array_intersect(array(1, 2, 3),array(1,3,4)); gives
> wrong output
> {noformat}
> [1,3] [1,3]{noformat}
> instead of
> {noformat}
> [1,2,3][1,3]{noformat}
> 2. SELECT array_intersect(array(),array()); should give empty array instead
> of NULL
> 3. Error handling when elements of array are not comparable
> These fixes were already present on array_except udf whose implementation is
> identical to array_intersect, Ref: [https://github.com/apache/hive/pull/4090]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27337) create and alter queries should trim empty spaces in the request objects.
[ https://issues.apache.org/jira/browse/HIVE-27337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17730804#comment-17730804 ] Sai Hemanth Gantasala commented on HIVE-27337: -- Thanks for the review [~dengzh]. Patch has been merged to the master branch. > create and alter queries should trim empty spaces in the request objects. > - > > Key: HIVE-27337 > URL: https://issues.apache.org/jira/browse/HIVE-27337 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 4.0.0-alpha-2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > If starting/trailing empty strings are being provided in the query (within > the quotes), > Create table request (event) is currently being inserted into the > notification log table. This would be a problem for listeners subscribed to > the HMS notification log. So it would be ideal to trim these whitespaces and > then insert it into the notification log. > Alter table query is throwing an error > [here|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java#L174]. > The ideal behavior should be to ignore the whitespaces and proceed with > alter request. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27337) create and alter queries should trim empty spaces in the request objects.
[ https://issues.apache.org/jira/browse/HIVE-27337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-27337. -- Fix Version/s: 4.0.0-alpha-1 Resolution: Fixed > create and alter queries should trim empty spaces in the request objects. > - > > Key: HIVE-27337 > URL: https://issues.apache.org/jira/browse/HIVE-27337 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 4.0.0-alpha-2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > If starting/trailing empty strings are being provided in the query (within > the quotes), > Create table request (event) is currently being inserted into the > notification log table. This would be a problem for listeners subscribed to > the HMS notification log. So it would be ideal to trim these whitespaces and > then insert it into the notification log. > Alter table query is throwing an error > [here|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java#L174]. > The ideal behavior should be to ignore the whitespaces and proceed with > alter request. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27319) HMS server should throw InvalidObjectException in get_partitions_by_names() when the table is missing/dropped
[ https://issues.apache.org/jira/browse/HIVE-27319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-27319. -- Resolution: Fixed > HMS server should throw InvalidObjectException in get_partitions_by_names() > when the table is missing/dropped > - > > Key: HIVE-27319 > URL: https://issues.apache.org/jira/browse/HIVE-27319 > Project: Hive > Issue Type: Bug > Components: Hive, Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > When the table object is dropped by a concurrent thread, the > get_partitions_by_names_req() API is currently throwing a > TApplicationException to the client. Instead, the HMS server should propagate > the InvalidObjectException thrown by getTable() to the HMS client. By doing > this, other services using HMS client will understand the exception better. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27319) HMS server should throw InvalidObjectException in get_partitions_by_names() when the table is missing/dropped
[ https://issues.apache.org/jira/browse/HIVE-27319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17725032#comment-17725032 ] Sai Hemanth Gantasala commented on HIVE-27319: -- Patch merged to the master branch. Thanks for the reviews [~dengzh] and [~ngangam] > HMS server should throw InvalidObjectException in get_partitions_by_names() > when the table is missing/dropped > - > > Key: HIVE-27319 > URL: https://issues.apache.org/jira/browse/HIVE-27319 > Project: Hive > Issue Type: Bug > Components: Hive, Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > When the table object is dropped by a concurrent thread, the > get_partitions_by_names_req() API is currently throwing a > TApplicationException to the client. Instead, the HMS server should propagate > the InvalidObjectException thrown by getTable() to the HMS client. By doing > this, other services using HMS client will understand the exception better. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27359) Backport HIVE-26850: ColumnInfo of the source table should not be null for the alter view as query to branch-3
Sai Hemanth Gantasala created HIVE-27359: Summary: Backport HIVE-26850: ColumnInfo of the source table should not be null for the alter view as query to branch-3 Key: HIVE-27359 URL: https://issues.apache.org/jira/browse/HIVE-27359 Project: Hive Issue Type: Sub-task Reporter: Sai Hemanth Gantasala Assignee: Sai Hemanth Gantasala The ColumnInfo for the source table of the alter view as query is coming as null and as a result, the column info is not present for the table. The ColumnInfo should be non-null. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27308) Exposing client keystore and truststore passwords in the JDBC URL can be a security concern
[
https://issues.apache.org/jira/browse/HIVE-27308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved HIVE-27308.
--
Fix Version/s: 4.0.0
Resolution: Fixed
> Exposing client keystore and truststore passwords in the JDBC URL can be a
> security concern
> ---
>
> Key: HIVE-27308
> URL: https://issues.apache.org/jira/browse/HIVE-27308
> Project: Hive
> Issue Type: Improvement
>Reporter: Venugopal Reddy K
>Assignee: Venugopal Reddy K
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> At present, we may have the following keystore and truststore passwords in
> the JDBC URL.
> # trustStorePassword
> # keyStorePassword
> # zooKeeperTruststorePassword
> # zooKeeperKeystorePassword
> Exposing these passwords in URL can be a security concern. Can hide all these
> passwords from JDBC URL when we protect these passwords in a local JCEKS
> keystore file and pass the JCEKS file to URL instead.
> 1. Leverage the hadoop credential provider
> [Link|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Overview]
> Create aliases for these passwords in a local JCE keystore like below. Store
> all the passwords in the same JCEKS files.
> {{hadoop credential create *keyStorePassword* -value
> FDUxmzTxW15xWoaCk6GxLlaoHjnjV9H7iHqCIDxTwoq -provider
> localjceks://file/tmp/store/client_creds.jceks}}
> 2. Add a new option *storePasswordPath* to JDBC URL that point to the local
> JCE keystore file storing the password aliases. When the existing password
> option is present in URL, can ignore to fetch that particular alias from
> local jceks(i.e., giving preference to existing password option). And if
> password option is not present in URL, can fetch the password from local
> jceks.
> JDBC URL may look like:
> {{beeline -u
> "jdbc:hive2://kvr-host:10001/default;retries=5;ssl=true;sslTrustStore=/tmp/truststore.jks;transportMode=http;httpPath=cliservice;twoWay=true;sslKeyStore=/tmp/keystore.jks;{*}storePasswordPath=localjceks://file/tmp/client_creds.jceks;{*}"}}
> 3. Hive JDBC can fetch the passwords with
> [Configuration.getPassword|https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/conf/Configuration.html#getPassword-java.lang.String-]
> API
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27308) Exposing client keystore and truststore passwords in the JDBC URL can be a security concern
[
https://issues.apache.org/jira/browse/HIVE-27308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17723538#comment-17723538
]
Sai Hemanth Gantasala commented on HIVE-27308:
--
[~VenuReddy] - Thanks for your contribution. The patch has been merged into the
master branch.
> Exposing client keystore and truststore passwords in the JDBC URL can be a
> security concern
> ---
>
> Key: HIVE-27308
> URL: https://issues.apache.org/jira/browse/HIVE-27308
> Project: Hive
> Issue Type: Improvement
>Reporter: Venugopal Reddy K
>Assignee: Venugopal Reddy K
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> At present, we may have the following keystore and truststore passwords in
> the JDBC URL.
> # trustStorePassword
> # keyStorePassword
> # zooKeeperTruststorePassword
> # zooKeeperKeystorePassword
> Exposing these passwords in URL can be a security concern. Can hide all these
> passwords from JDBC URL when we protect these passwords in a local JCEKS
> keystore file and pass the JCEKS file to URL instead.
> 1. Leverage the hadoop credential provider
> [Link|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Overview]
> Create aliases for these passwords in a local JCE keystore like below. Store
> all the passwords in the same JCEKS files.
> {{hadoop credential create *keyStorePassword* -value
> FDUxmzTxW15xWoaCk6GxLlaoHjnjV9H7iHqCIDxTwoq -provider
> localjceks://file/tmp/store/client_creds.jceks}}
> 2. Add a new option *storePasswordPath* to JDBC URL that point to the local
> JCE keystore file storing the password aliases. When the existing password
> option is present in URL, can ignore to fetch that particular alias from
> local jceks(i.e., giving preference to existing password option). And if
> password option is not present in URL, can fetch the password from local
> jceks.
> JDBC URL may look like:
> {{beeline -u
> "jdbc:hive2://kvr-host:10001/default;retries=5;ssl=true;sslTrustStore=/tmp/truststore.jks;transportMode=http;httpPath=cliservice;twoWay=true;sslKeyStore=/tmp/keystore.jks;{*}storePasswordPath=localjceks://file/tmp/client_creds.jceks;{*}"}}
> 3. Hive JDBC can fetch the passwords with
> [Configuration.getPassword|https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/conf/Configuration.html#getPassword-java.lang.String-]
> API
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27337) create and alter queries should trim empty spaces in the request objects.
Sai Hemanth Gantasala created HIVE-27337: Summary: create and alter queries should trim empty spaces in the request objects. Key: HIVE-27337 URL: https://issues.apache.org/jira/browse/HIVE-27337 Project: Hive Issue Type: Bug Components: Standalone Metastore Affects Versions: 4.0.0-alpha-2 Reporter: Sai Hemanth Gantasala Assignee: Sai Hemanth Gantasala If starting/trailing empty strings are being provided in the query (within the quotes), Create table request (event) is currently being inserted into the notification log table. This would be a problem for listeners subscribed to the HMS notification log. So it would be ideal to trim these whitespaces and then insert it into the notification log. Alter table query is throwing an error [here|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java#L174]. The ideal behavior should be to ignore the whitespaces and proceed with alter request. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27268) Hive.getPartitionsByNames should not enforce SessionState to be available
[ https://issues.apache.org/jira/browse/HIVE-27268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721072#comment-17721072 ] Sai Hemanth Gantasala commented on HIVE-27268: -- [~henrib] - Thanks for your contribution. Fix has been merged into the master branch. > Hive.getPartitionsByNames should not enforce SessionState to be available > - > > Key: HIVE-27268 > URL: https://issues.apache.org/jira/browse/HIVE-27268 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.3 >Reporter: Henri Biestro >Assignee: Henri Biestro >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > HIVE-24743, HIVE-24392 is enforcing to check for valid write Id list for > "Hive.getPartitionsByName". > This breaks basic API integration. For a user who needs to get basic > partition detail, he is forced to have SessionState. > Request in this ticket is to ensure that if SessionState.get() is null, it > should return empty validWriteIdList. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27268) Hive.getPartitionsByNames should not enforce SessionState to be available
[ https://issues.apache.org/jira/browse/HIVE-27268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-27268. -- Resolution: Fixed > Hive.getPartitionsByNames should not enforce SessionState to be available > - > > Key: HIVE-27268 > URL: https://issues.apache.org/jira/browse/HIVE-27268 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.3 >Reporter: Henri Biestro >Assignee: Henri Biestro >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > HIVE-24743, HIVE-24392 is enforcing to check for valid write Id list for > "Hive.getPartitionsByName". > This breaks basic API integration. For a user who needs to get basic > partition detail, he is forced to have SessionState. > Request in this ticket is to ensure that if SessionState.get() is null, it > should return empty validWriteIdList. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27268) Hive.getPartitionsByNames should not enforce SessionState to be available
[ https://issues.apache.org/jira/browse/HIVE-27268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala reassigned HIVE-27268: Assignee: Henri Biestro > Hive.getPartitionsByNames should not enforce SessionState to be available > - > > Key: HIVE-27268 > URL: https://issues.apache.org/jira/browse/HIVE-27268 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.3 >Reporter: Henri Biestro >Assignee: Henri Biestro >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > HIVE-24743, HIVE-24392 is enforcing to check for valid write Id list for > "Hive.getPartitionsByName". > This breaks basic API integration. For a user who needs to get basic > partition detail, he is forced to have SessionState. > Request in this ticket is to ensure that if SessionState.get() is null, it > should return empty validWriteIdList. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27319) HMS server should throw InvalidObjectException in get_partitions_by_names() when the table is missing/dropped
[ https://issues.apache.org/jira/browse/HIVE-27319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala updated HIVE-27319: - Summary: HMS server should throw InvalidObjectException in get_partitions_by_names() when the table is missing/dropped (was: HMS server should throw InvalidObjectException when the table is missing in get_partitions_by_names()) > HMS server should throw InvalidObjectException in get_partitions_by_names() > when the table is missing/dropped > - > > Key: HIVE-27319 > URL: https://issues.apache.org/jira/browse/HIVE-27319 > Project: Hive > Issue Type: Bug > Components: Hive, Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When the table object is dropped by a concurrent thread, the > get_partitions_by_names_req() API is currently throwing a > TApplicationException to the client. Instead, the HMS server should propagate > the InvalidObjectException thrown by getTable() to the HMS client. By doing > this, other services using HMS client will understand the exception better. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27319) HMS server should throw InvalidObjectException when the table is missing in get_partitions_by_names()
Sai Hemanth Gantasala created HIVE-27319: Summary: HMS server should throw InvalidObjectException when the table is missing in get_partitions_by_names() Key: HIVE-27319 URL: https://issues.apache.org/jira/browse/HIVE-27319 Project: Hive Issue Type: Bug Components: Hive, Standalone Metastore Reporter: Sai Hemanth Gantasala Assignee: Sai Hemanth Gantasala When the table object is dropped by a concurrent thread, the get_partitions_by_names_req() API is currently throwing a TApplicationException to the client. Instead, the HMS server should propagate the InvalidObjectException thrown by getTable() to the HMS client. By doing this, other services using HMS client will understand the exception better. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-25114) Optmize get_tables() api call in HMS
[ https://issues.apache.org/jira/browse/HIVE-25114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-25114. -- Resolution: Won't Fix > Optmize get_tables() api call in HMS > > > Key: HIVE-25114 > URL: https://issues.apache.org/jira/browse/HIVE-25114 > Project: Hive > Issue Type: Improvement >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Optimize get_tables() call in HMS API. There should be batching in the > get_tables() APIs when it is fetching table names as a list in the arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27286) Upgrade jettison version to 1.5.4 to address CVE-2023-1436
[ https://issues.apache.org/jira/browse/HIVE-27286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17718666#comment-17718666 ] Sai Hemanth Gantasala commented on HIVE-27286: -- [~ssand] - Thanks for your contribution. The patch has been merged into the master branch. > Upgrade jettison version to 1.5.4 to address CVE-2023-1436 > -- > > Key: HIVE-27286 > URL: https://issues.apache.org/jira/browse/HIVE-27286 > Project: Hive > Issue Type: Improvement >Reporter: Sand Shreeya >Assignee: Sand Shreeya >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27285) Add TableMeta ownership for filterTableMetas
[
https://issues.apache.org/jira/browse/HIVE-27285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved HIVE-27285.
--
Fix Version/s: 4.0.0-alpha-2
Resolution: Fixed
> Add TableMeta ownership for filterTableMetas
>
>
> Key: HIVE-27285
> URL: https://issues.apache.org/jira/browse/HIVE-27285
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Currently TableMeta does not include ownership information which makes it
> difficult for filterTableMetas to efficiently filter based on ${OWNER}
> privileges.
> We should add ownership information to TableMeta and utilizing it in
> filterTableMetas authorization checks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27285) Add TableMeta ownership for filterTableMetas
[
https://issues.apache.org/jira/browse/HIVE-27285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17718349#comment-17718349
]
Sai Hemanth Gantasala commented on HIVE-27285:
--
[~jfs] Thanks for your contribution. Patch merged to master.
> Add TableMeta ownership for filterTableMetas
>
>
> Key: HIVE-27285
> URL: https://issues.apache.org/jira/browse/HIVE-27285
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Currently TableMeta does not include ownership information which makes it
> difficult for filterTableMetas to efficiently filter based on ${OWNER}
> privileges.
> We should add ownership information to TableMeta and utilizing it in
> filterTableMetas authorization checks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-26838) Add a new event to improve cache performance in external systems that communicates with HMS.
[ https://issues.apache.org/jira/browse/HIVE-26838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-26838. -- Fix Version/s: 4.0.0 Resolution: Fixed > Add a new event to improve cache performance in external systems that > communicates with HMS. > > > Key: HIVE-26838 > URL: https://issues.apache.org/jira/browse/HIVE-26838 > Project: Hive > Issue Type: New Feature > Components: Hive, Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Adding support for a new event "Reload event" in the HMS (HiveMetaStore). > This event can be used by external services that depend on HMS for metadata > operations to improve its cache performance. In the distributed environment > where there are replicas of an external service (with its own cache in each > of these replicas) talking to HMS for metadata operations, the reload event > can be used to address the cache performance and ensure consistency among all > the replicas for a given table/partition. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26838) Add a new event to improve cache performance in external systems that communicates with HMS.
[ https://issues.apache.org/jira/browse/HIVE-26838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717390#comment-17717390 ] Sai Hemanth Gantasala commented on HIVE-26838: -- Patch merged to master. [~dengzh] thanks for the review. > Add a new event to improve cache performance in external systems that > communicates with HMS. > > > Key: HIVE-26838 > URL: https://issues.apache.org/jira/browse/HIVE-26838 > Project: Hive > Issue Type: New Feature > Components: Hive, Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > Adding support for a new event "Reload event" in the HMS (HiveMetaStore). > This event can be used by external services that depend on HMS for metadata > operations to improve its cache performance. In the distributed environment > where there are replicas of an external service (with its own cache in each > of these replicas) talking to HMS for metadata operations, the reload event > can be used to address the cache performance and ensure consistency among all > the replicas for a given table/partition. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27304) Exclude CTAS condition while forming storage handler url permissions in HS2 authorizer.
Sai Hemanth Gantasala created HIVE-27304: Summary: Exclude CTAS condition while forming storage handler url permissions in HS2 authorizer. Key: HIVE-27304 URL: https://issues.apache.org/jira/browse/HIVE-27304 Project: Hive Issue Type: Bug Components: Hive Reporter: Sai Hemanth Gantasala Assignee: Sai Hemanth Gantasala CTAS queries with storage handler table as source table doesn't require read permissions on storage URL for the target table -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27257) Database object is missing owner info in the CreateDatabaseRequest
[
https://issues.apache.org/jira/browse/HIVE-27257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved HIVE-27257.
--
Resolution: Not A Problem
> Database object is missing owner info in the CreateDatabaseRequest
> --
>
> Key: HIVE-27257
> URL: https://issues.apache.org/jira/browse/HIVE-27257
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
>
> CreateDatabaseRequest should have owner info so that \{OWNER} policies are
> honoured in authorization services like ranger/sentry.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27186) A persistent property store
[ https://issues.apache.org/jira/browse/HIVE-27186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala reassigned HIVE-27186: Assignee: Henri Biestro > A persistent property store > > > Key: HIVE-27186 > URL: https://issues.apache.org/jira/browse/HIVE-27186 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 4.0.0-alpha-2 >Reporter: Henri Biestro >Assignee: Henri Biestro >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > WHAT > A persistent property store usable as a support facility for any metadata > augmentation feature. > WHY > When adding new meta-data oriented features, we usually need to persist > information linking the feature data and the HiveMetaStore objects it applies > to. Any information related to a database, a table or the cluster - like > statistics for example or any operational data state or data (think rolling > backup) - fall in this use-case. > Typically, accommodating such a feature requires modifying the Metastore > database schema by adding or altering a table. It also usually implies > modifying the thrift APIs to expose such meta-data to consumers. > The proposed feature wants to solve the persistence and query/transport for > these types of use-cases by exposing a 'key/(meta)value' store exposed as a > property system. > HOW > A property-value model is the simple and generic exposed API. > To provision for several usage scenarios, the model entry point is a > 'namespace' that qualifies the feature-component property manager. For > example, 'stats' could be the namespace for all properties related to the > 'statistics' feature. > The namespace identifies a manager that handles property-groups persisted as > property-maps. For instance, all statistics pertaining to a given table would > be collocated in the same property-group. As such, all properties (say number > of 'unique_values' per columns) for a given HMS table 'relation0' would all > be stored and persisted in the same property-map instance. > Property-maps may be decorated by an (optional) schema that may declare the > name and value-type of allowed properties (and their optional default value). > Each property is addressed by a name, a path uniquely identifying the > property in a given property map. > The manager also handles transforming property-map names to the property-map > keys used to persist them in the DB. > The API provides inserting/updating properties in bulk transactionally. It > also provides selection/projection to help reduce the volume of exchange > between client/server; selection can use (JEXL expression) predicates to > filter maps. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27257) Database object is missing owner info in the CreateDatabaseRequest
Sai Hemanth Gantasala created HIVE-27257:
Summary: Database object is missing owner info in the
CreateDatabaseRequest
Key: HIVE-27257
URL: https://issues.apache.org/jira/browse/HIVE-27257
Project: Hive
Issue Type: Bug
Components: HiveServer2
Reporter: Sai Hemanth Gantasala
Assignee: Sai Hemanth Gantasala
CreateDatabaseRequest should have owner info so that \{OWNER} policies are
honoured in authorization services like ranger/sentry.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27146) Re-enable orc_merge*.q tests for TestMiniSparkOnYarnCliDriver
[
https://issues.apache.org/jira/browse/HIVE-27146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704063#comment-17704063
]
Sai Hemanth Gantasala commented on HIVE-27146:
--
[~vihangk1] - Do you want to mask the HDFS path if these qfiles are relatively
smaller in number?
If have previously masked the total size of the table parameter using the
following config in the qfile
{code:java}
--! qt:replace:/(\s+totalSize\s+)\S+(\s+)/$1#Masked#$2/ {code}
> Re-enable orc_merge*.q tests for TestMiniSparkOnYarnCliDriver
> -
>
> Key: HIVE-27146
> URL: https://issues.apache.org/jira/browse/HIVE-27146
> Project: Hive
> Issue Type: Test
>Affects Versions: 3.1.3, 3.2.0
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> It was found that the q.out file for these tests fail with a diff in the
> replication factor of the files. The tests only fail on the CI job so it is
> possible that it is due to some test environment issues. The tests also fail
> on 3.1.3 release.
> E.g orc_merge4.q fails with the error. Similarly the other tests fail with
> the same difference in replication factor.
> {code:java}
> 40c40
> < -rw-r--r-- 1 ### USER ### ### GROUP ### 2530 ### HDFS DATE ###
> hdfs://### HDFS PATH ###
> ---
> > -rw-r--r-- 3 ### USER ### ### GROUP ### 2530 ### HDFS DATE ###
> > hdfs://### HDFS PATH ###
> 66c66
> < -rw-r--r-- 1 ### USER ### ### GROUP ### 2530 ### HDFS DATE ###
> hdfs://### HDFS PATH ###
> ---
> > -rw-r--r-- 3 ### USER ### ### GROUP ### 2530 ### HDFS DATE ###
> > hdfs://### HDFS PATH ###
> 68c68
> < -rw-r--r-- 1 ### USER ### ### GROUP ### 2530 ### HDFS DATE ###
> hdfs://### HDFS PATH ###
> ---
> > -rw-r--r-- 3 ### USER ### ### GROUP ### 2530 ### HDFS DATE ###
> > hdfs://### HDFS PATH ###
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27116) HS2 need to send owner info for UDFs in the HivePrivilegeObject for authorization
[ https://issues.apache.org/jira/browse/HIVE-27116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala updated HIVE-27116: - Description: UDFs (Functions) of HivePrivilegeObject should have owner info for authorization. (was: UDFs (Functions) of HivePrivilegeObject should send owner info for authorization.) > HS2 need to send owner info for UDFs in the HivePrivilegeObject for > authorization > - > > Key: HIVE-27116 > URL: https://issues.apache.org/jira/browse/HIVE-27116 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Security >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > > UDFs (Functions) of HivePrivilegeObject should have owner info for > authorization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27116) HS2 need to send owner info for UDFs in the HivePrivilegeObject for authorization
[ https://issues.apache.org/jira/browse/HIVE-27116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala updated HIVE-27116: - Summary: HS2 need to send owner info for UDFs in the HivePrivilegeObject for authorization (was: HS2 need to send owner info in the HivePrivilegeObject for authorization) > HS2 need to send owner info for UDFs in the HivePrivilegeObject for > authorization > - > > Key: HIVE-27116 > URL: https://issues.apache.org/jira/browse/HIVE-27116 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Security >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > > UDFs (Functions) of HivePrivilegeObject should send owner info for > authorization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27116) HS2 need to send owner info in the HivePrivilegeObject for authorization
[ https://issues.apache.org/jira/browse/HIVE-27116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala reassigned HIVE-27116: > HS2 need to send owner info in the HivePrivilegeObject for authorization > > > Key: HIVE-27116 > URL: https://issues.apache.org/jira/browse/HIVE-27116 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Security >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > > UDFs (Functions) of HivePrivilegeObject should send owner info for > authorization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27091) Add double quotes for tables in PartitionProjectionEvaluator
[
https://issues.apache.org/jira/browse/HIVE-27091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17694592#comment-17694592
]
Sai Hemanth Gantasala commented on HIVE-27091:
--
Thanks for the contribution [~dengzh]. Patch has been merged to master.
> Add double quotes for tables in PartitionProjectionEvaluator
>
>
> Key: HIVE-27091
> URL: https://issues.apache.org/jira/browse/HIVE-27091
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When PartitionProjectionEvaluator requests partitions against PostgreSQL,
> there throws exception:
> {noformat}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "SDS"."LOCATION","PARTITIONS"."CREATE_TIME","SDS"."SD_ID","PARTITIONS"."PART_ID"
> from PARTITIONS left outer join SDS on PARTITIONS."SD_ID" = SDS."SD_ID"
> left outer join SERDES on SDS."SERDE_ID" = SERDES."SERDE_ID" where "PART_ID"
> in (92731,92732,92733,92734,92735,92736) order by "PART_NAME" asc".
> …
> Caused by: org.postgresql.util.PSQLException: ERROR: relation "partitions"
> does not exist{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27071) Select query with LIMIT clause can fail if there are marker files like "_SUCCESS" and "_MANIFEST"
[
https://issues.apache.org/jira/browse/HIVE-27071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688037#comment-17688037
]
Sai Hemanth Gantasala commented on HIVE-27071:
--
[~ayushtkn] - Thanks for providing the patch in a quick time. I have reviewed
the PR. I'll assign this Jira to you since you are the author of this patch.
[~tarak271] - I'm not sure what other files we want to skip other than the
names starting with "_" or ".". Can you suggest any other files? Maybe Ayush
has a better idea if it is good to skip those unwanted files.
> Select query with LIMIT clause can fail if there are marker files like
> "_SUCCESS" and "_MANIFEST"
> -
>
> Key: HIVE-27071
> URL: https://issues.apache.org/jira/browse/HIVE-27071
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Spark clients creates marker files like "_SUCCESS" and "_MANIFEST" under the
> table/partition path at the end of a write operation. For example
> 'hdfs://name-node-host/table/partition/_SUCCESS'
> Whenever Hive is trying to read that table with the LIMIT clause, it could to
> the following error:
> {code:java}
> ERROR : Vertex failed, vertexName=Map 1,
> vertexId=vertex_1676095298574_0017_2_00, diagnostics=[Vertex
> vertex_1676095298574_0017_2_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: trade initializer failed,
> vertex=vertex_1676095298574_0017_2_00 [Map 1],
> org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:
> hdfs://name-node-host/table/partition/_MANIFEST
> Input path does not exist: hdfs://name-node-host/table/partition/_SUCCESS at
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:300)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:240)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:328)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:579)
> {code}
> Hive execution engine should ignore these marker files while reading the
> table/partition data.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27071) Select query with LIMIT clause can fail if there are marker files like "_SUCCESS" and "_MANIFEST"
[
https://issues.apache.org/jira/browse/HIVE-27071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala reassigned HIVE-27071:
Assignee: Ayush Saxena
> Select query with LIMIT clause can fail if there are marker files like
> "_SUCCESS" and "_MANIFEST"
> -
>
> Key: HIVE-27071
> URL: https://issues.apache.org/jira/browse/HIVE-27071
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Sai Hemanth Gantasala
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Spark clients creates marker files like "_SUCCESS" and "_MANIFEST" under the
> table/partition path at the end of a write operation. For example
> 'hdfs://name-node-host/table/partition/_SUCCESS'
> Whenever Hive is trying to read that table with the LIMIT clause, it could to
> the following error:
> {code:java}
> ERROR : Vertex failed, vertexName=Map 1,
> vertexId=vertex_1676095298574_0017_2_00, diagnostics=[Vertex
> vertex_1676095298574_0017_2_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: trade initializer failed,
> vertex=vertex_1676095298574_0017_2_00 [Map 1],
> org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:
> hdfs://name-node-host/table/partition/_MANIFEST
> Input path does not exist: hdfs://name-node-host/table/partition/_SUCCESS at
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:300)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:240)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:328)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:579)
> {code}
> Hive execution engine should ignore these marker files while reading the
> table/partition data.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27071) Select query with LIMIT clause can fail if their are marker files like "_SUCCESS" and "_MANIFEST"
[
https://issues.apache.org/jira/browse/HIVE-27071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala reassigned HIVE-27071:
> Select query with LIMIT clause can fail if their are marker files like
> "_SUCCESS" and "_MANIFEST"
> -
>
> Key: HIVE-27071
> URL: https://issues.apache.org/jira/browse/HIVE-27071
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Sai Hemanth Gantasala
>Assignee: László Bodor
>Priority: Major
>
> Spark clients creates marker files like "_SUCCESS" and "_MANIFEST" under the
> table/partition path at the end of a write operation. For example
> 'hdfs://name-node-host/table/partition/_SUCCESS'
> Whenever Hive is trying to read that table with the LIMIT clause, it could to
> the following error:
> {code:java}
> ERROR : Vertex failed, vertexName=Map 1,
> vertexId=vertex_1676095298574_0017_2_00, diagnostics=[Vertex
> vertex_1676095298574_0017_2_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: trade initializer failed,
> vertex=vertex_1676095298574_0017_2_00 [Map 1],
> org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:
> hdfs://name-node-host/table/partition/_MANIFEST
> Input path does not exist: hdfs://name-node-host/table/partition/_SUCCESS at
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:300)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:240)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:328)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:579)
> {code}
> Hive execution engine should ignore these marker files while reading the
> table/partition data.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27071) Select query with LIMIT clause can fail if there are marker files like "_SUCCESS" and "_MANIFEST"
[
https://issues.apache.org/jira/browse/HIVE-27071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala updated HIVE-27071:
-
Summary: Select query with LIMIT clause can fail if there are marker files
like "_SUCCESS" and "_MANIFEST" (was: Select query with LIMIT clause can fail
if their are marker files like "_SUCCESS" and "_MANIFEST")
> Select query with LIMIT clause can fail if there are marker files like
> "_SUCCESS" and "_MANIFEST"
> -
>
> Key: HIVE-27071
> URL: https://issues.apache.org/jira/browse/HIVE-27071
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Sai Hemanth Gantasala
>Assignee: László Bodor
>Priority: Major
>
> Spark clients creates marker files like "_SUCCESS" and "_MANIFEST" under the
> table/partition path at the end of a write operation. For example
> 'hdfs://name-node-host/table/partition/_SUCCESS'
> Whenever Hive is trying to read that table with the LIMIT clause, it could to
> the following error:
> {code:java}
> ERROR : Vertex failed, vertexName=Map 1,
> vertexId=vertex_1676095298574_0017_2_00, diagnostics=[Vertex
> vertex_1676095298574_0017_2_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: trade initializer failed,
> vertex=vertex_1676095298574_0017_2_00 [Map 1],
> org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:
> hdfs://name-node-host/table/partition/_MANIFEST
> Input path does not exist: hdfs://name-node-host/table/partition/_SUCCESS at
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:300)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:240)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:328)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:579)
> {code}
> Hive execution engine should ignore these marker files while reading the
> table/partition data.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27004) DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions higher than 8
[
https://issues.apache.org/jira/browse/HIVE-27004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683656#comment-17683656
]
Sai Hemanth Gantasala commented on HIVE-27004:
--
The fix has been merged to the master. Thanks for your contribution [~AnmolSun]
> DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions higher than 8
> ---
>
> Key: HIVE-27004
> URL: https://issues.apache.org/jira/browse/HIVE-27004
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Environment: Java version used
> {code:java}
> openjdk version "11.0.16.1" 2022-08-12 LTS OpenJDK Runtime Environment
> Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS) OpenJDK 64-Bit Server VM
> Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS, mixed mode{code}
>
>Reporter: Anmol Sundaram
>Assignee: Anmol Sundaram
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-27004.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Some of the unit tests related to _DateTimeFormatter_ were failing in Java
> versions greater than 8 while working in Java 8.
> Example of a failing Unit Test :
> _org.apache.hadoop.hive.common.type.TestTimestampTZ#testComparision_
>
> {code:java}
> java.time.format.DateTimeParseException: Text '2017-04-14 18:00:00 UTC+08:00'
> could not be parsed, unparsed text found at index 23 at
> java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2049)
> at
> java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1874)
> at
> org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:76)
> at
> org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:64)
> at
> org.apache.hadoop.hive.common.type.TestTimestampTZ.testComparison(TestTimestampTZ.java:44)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method){code}
>
> The {{appendZoneText(TextStyle)}} method of
> [DateTimeFormatteBuilder|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/type/TimestampTZUtil.java#L82]
> is not able to parse the {{+}} symbol in {{2017-04-14 18:00:00 UTC+08:00}}
> when running [the
> test|https://github.com/apache/hive/blob/master/common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java#L37]
> in Java 11 , while it is working fine with Java 8.
> According to the
> [doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendZoneText(java.time.format.TextStyle)]
> , {{appendZoneText(TextStyle)}} should be able to parse {{either the textual
> zone name, the zone ID or the offset}} and UTC+08:00 should come under offset
> as per the same
> [doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendOffset(java.lang.String,%20java.lang.String)].
> It seems, however, that "UTC" was explicitly removed [when parsed as ZoneText
> for higher Java
> versions|https://github.com/openjdk/jdk/commit/5c3a01591c5c945926636fdc9f164d60b5b4f29e?diff=unified#diff-5fcf976db1c06e8f44a8671356d7e34fdfbf5b057baa852e7c3e015c8797c889R4263]
> As a workaround, we can use {{appendZoneOrOffsetId()}} rather than
> {{appendZoneText().}}
> This ensures the tests are passing for Java 8+ and based on my testing, I
> didn’t see any regression of the change.
> Sample repro code - jdoodle.com/ia/D5e
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27004) DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions higher than 8
[
https://issues.apache.org/jira/browse/HIVE-27004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala updated HIVE-27004:
-
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions higher than 8
> ---
>
> Key: HIVE-27004
> URL: https://issues.apache.org/jira/browse/HIVE-27004
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Environment: Java version used
> {code:java}
> openjdk version "11.0.16.1" 2022-08-12 LTS OpenJDK Runtime Environment
> Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS) OpenJDK 64-Bit Server VM
> Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS, mixed mode{code}
>
>Reporter: Anmol Sundaram
>Assignee: Anmol Sundaram
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-27004.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Some of the unit tests related to _DateTimeFormatter_ were failing in Java
> versions greater than 8 while working in Java 8.
> Example of a failing Unit Test :
> _org.apache.hadoop.hive.common.type.TestTimestampTZ#testComparision_
>
> {code:java}
> java.time.format.DateTimeParseException: Text '2017-04-14 18:00:00 UTC+08:00'
> could not be parsed, unparsed text found at index 23 at
> java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2049)
> at
> java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1874)
> at
> org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:76)
> at
> org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:64)
> at
> org.apache.hadoop.hive.common.type.TestTimestampTZ.testComparison(TestTimestampTZ.java:44)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method){code}
>
> The {{appendZoneText(TextStyle)}} method of
> [DateTimeFormatteBuilder|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/type/TimestampTZUtil.java#L82]
> is not able to parse the {{+}} symbol in {{2017-04-14 18:00:00 UTC+08:00}}
> when running [the
> test|https://github.com/apache/hive/blob/master/common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java#L37]
> in Java 11 , while it is working fine with Java 8.
> According to the
> [doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendZoneText(java.time.format.TextStyle)]
> , {{appendZoneText(TextStyle)}} should be able to parse {{either the textual
> zone name, the zone ID or the offset}} and UTC+08:00 should come under offset
> as per the same
> [doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendOffset(java.lang.String,%20java.lang.String)].
> It seems, however, that "UTC" was explicitly removed [when parsed as ZoneText
> for higher Java
> versions|https://github.com/openjdk/jdk/commit/5c3a01591c5c945926636fdc9f164d60b5b4f29e?diff=unified#diff-5fcf976db1c06e8f44a8671356d7e34fdfbf5b057baa852e7c3e015c8797c889R4263]
> As a workaround, we can use {{appendZoneOrOffsetId()}} rather than
> {{appendZoneText().}}
> This ensures the tests are passing for Java 8+ and based on my testing, I
> didn’t see any regression of the change.
> Sample repro code - jdoodle.com/ia/D5e
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26889) Implement array_join udf to concatenate the elements of an array with a specified delimiter
[
https://issues.apache.org/jira/browse/HIVE-26889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683508#comment-17683508
]
Sai Hemanth Gantasala commented on HIVE-26889:
--
Fix has been merged to the master. Thanks for your contribution [~tarak271]
> Implement array_join udf to concatenate the elements of an array with a
> specified delimiter
> ---
>
> Key: HIVE-26889
> URL: https://issues.apache.org/jira/browse/HIVE-26889
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> *array_join(array, delimiter, replaceNull)* - concatenate the elements of an
> array with a specified delimiter
> Example:
> {noformat}
> > SELECT array_join(array(1, 2, 3,4), ',') FROM src LIMIT 1;
> 1,2,3,4
> > SELECT array_join(array(1, 2, NULL, 4), ',',':') FROM src LIMIT 1;
> 1,2,:,4{noformat}
> Returns NULL if array is null/empty
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-26889) Implement array_join udf to concatenate the elements of an array with a specified delimiter
[
https://issues.apache.org/jira/browse/HIVE-26889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved HIVE-26889.
--
Fix Version/s: 4.0.0
Resolution: Fixed
> Implement array_join udf to concatenate the elements of an array with a
> specified delimiter
> ---
>
> Key: HIVE-26889
> URL: https://issues.apache.org/jira/browse/HIVE-26889
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> *array_join(array, delimiter, replaceNull)* - concatenate the elements of an
> array with a specified delimiter
> Example:
> {noformat}
> > SELECT array_join(array(1, 2, 3,4), ',') FROM src LIMIT 1;
> 1,2,3,4
> > SELECT array_join(array(1, 2, NULL, 4), ',',':') FROM src LIMIT 1;
> 1,2,:,4{noformat}
> Returns NULL if array is null/empty
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27004) DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions higher than 8
[
https://issues.apache.org/jira/browse/HIVE-27004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683115#comment-17683115
]
Sai Hemanth Gantasala commented on HIVE-27004:
--
[~AnmolSun] - Can you upload your patch in the form of GitHub PR? That would be
easier to review and merge the patch.
> DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions higher than 8
> ---
>
> Key: HIVE-27004
> URL: https://issues.apache.org/jira/browse/HIVE-27004
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Environment: Java version used
> {code:java}
> openjdk version "11.0.16.1" 2022-08-12 LTS OpenJDK Runtime Environment
> Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS) OpenJDK 64-Bit Server VM
> Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS, mixed mode{code}
>
>Reporter: Anmol Sundaram
>Assignee: Anmol Sundaram
>Priority: Minor
> Attachments: HIVE-27004.patch
>
>
> Some of the unit tests related to _DateTimeFormatter_ were failing in Java
> versions greater than 8 while working in Java 8.
> Example of a failing Unit Test :
> _org.apache.hadoop.hive.common.type.TestTimestampTZ#testComparision_
>
> {code:java}
> java.time.format.DateTimeParseException: Text '2017-04-14 18:00:00 UTC+08:00'
> could not be parsed, unparsed text found at index 23 at
> java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2049)
> at
> java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1874)
> at
> org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:76)
> at
> org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:64)
> at
> org.apache.hadoop.hive.common.type.TestTimestampTZ.testComparison(TestTimestampTZ.java:44)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method){code}
>
> The {{appendZoneText(TextStyle)}} method of
> [DateTimeFormatteBuilder|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/type/TimestampTZUtil.java#L82]
> is not able to parse the {{+}} symbol in {{2017-04-14 18:00:00 UTC+08:00}}
> when running [the
> test|https://github.com/apache/hive/blob/master/common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java#L37]
> in Java 11 , while it is working fine with Java 8.
> According to the
> [doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendZoneText(java.time.format.TextStyle)]
> , {{appendZoneText(TextStyle)}} should be able to parse {{either the textual
> zone name, the zone ID or the offset}} and UTC+08:00 should come under offset
> as per the same
> [doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendOffset(java.lang.String,%20java.lang.String)].
> It seems, however, that "UTC" was explicitly removed [when parsed as ZoneText
> for higher Java
> versions|https://github.com/openjdk/jdk/commit/5c3a01591c5c945926636fdc9f164d60b5b4f29e?diff=unified#diff-5fcf976db1c06e8f44a8671356d7e34fdfbf5b057baa852e7c3e015c8797c889R4263]
> As a workaround, we can use {{appendZoneOrOffsetId()}} rather than
> {{appendZoneText().}}
> This ensures the tests are passing for Java 8+ and based on my testing, I
> didn’t see any regression of the change.
> Sample repro code - jdoodle.com/ia/D5e
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-26793) Create a new configuration to override "no compaction" for tables
[
https://issues.apache.org/jira/browse/HIVE-26793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved HIVE-26793.
--
Fix Version/s: 4.0.0
Resolution: Fixed
> Create a new configuration to override "no compaction" for tables
> -
>
> Key: HIVE-26793
> URL: https://issues.apache.org/jira/browse/HIVE-26793
> Project: Hive
> Issue Type: Improvement
>Reporter: Kokila N
>Assignee: Kokila N
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> Currently a simple user can create a table with
> {color:#6a8759}no_auto_compaction=true{color} table property and create an
> aborted write transaction writing to this table. This way a malicious user
> can prevent cleaning up data for the aborted transaction, creating
> performance degradation.
> This configuration should be allowed to overridden on a database level:
> adding {color:#6a8759}no_auto_compaction=false{color} should override the
> table level setting forcing the initiator to schedule compaction for all
> tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26793) Create a new configuration to override "no compaction" for tables
[
https://issues.apache.org/jira/browse/HIVE-26793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682752#comment-17682752
]
Sai Hemanth Gantasala commented on HIVE-26793:
--
Fix has been merged to the master. Thanks for your contribution [~kokila19]
> Create a new configuration to override "no compaction" for tables
> -
>
> Key: HIVE-26793
> URL: https://issues.apache.org/jira/browse/HIVE-26793
> Project: Hive
> Issue Type: Improvement
>Reporter: Kokila N
>Assignee: Kokila N
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> Currently a simple user can create a table with
> {color:#6a8759}no_auto_compaction=true{color} table property and create an
> aborted write transaction writing to this table. This way a malicious user
> can prevent cleaning up data for the aborted transaction, creating
> performance degradation.
> This configuration should be allowed to overridden on a database level:
> adding {color:#6a8759}no_auto_compaction=false{color} should override the
> table level setting forcing the initiator to schedule compaction for all
> tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27004) DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions higher than 8
[
https://issues.apache.org/jira/browse/HIVE-27004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682605#comment-17682605
]
Sai Hemanth Gantasala commented on HIVE-27004:
--
cc [~zabetak] Can you review this patch?
> DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions higher than 8
> ---
>
> Key: HIVE-27004
> URL: https://issues.apache.org/jira/browse/HIVE-27004
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Environment: Java version used
> {code:java}
> openjdk version "11.0.16.1" 2022-08-12 LTS OpenJDK Runtime Environment
> Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS) OpenJDK 64-Bit Server VM
> Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS, mixed mode{code}
>
>Reporter: Anmol Sundaram
>Assignee: Anmol Sundaram
>Priority: Minor
> Attachments: HIVE-27004.patch
>
>
> Some of the unit tests related to _DateTimeFormatter_ were failing in Java
> versions greater than 8 while working in Java 8.
> Example of a failing Unit Test :
> _org.apache.hadoop.hive.common.type.TestTimestampTZ#testComparision_
>
> {code:java}
> java.time.format.DateTimeParseException: Text '2017-04-14 18:00:00 UTC+08:00'
> could not be parsed, unparsed text found at index 23 at
> java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2049)
> at
> java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1874)
> at
> org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:76)
> at
> org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:64)
> at
> org.apache.hadoop.hive.common.type.TestTimestampTZ.testComparison(TestTimestampTZ.java:44)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method){code}
>
> The {{appendZoneText(TextStyle)}} method of
> [DateTimeFormatteBuilder|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/type/TimestampTZUtil.java#L82]
> is not able to parse the {{+}} symbol in {{2017-04-14 18:00:00 UTC+08:00}}
> when running [the
> test|https://github.com/apache/hive/blob/master/common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java#L37]
> in Java 11 , while it is working fine with Java 8.
> According to the
> [doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendZoneText(java.time.format.TextStyle)]
> , {{appendZoneText(TextStyle)}} should be able to parse {{either the textual
> zone name, the zone ID or the offset}} and UTC+08:00 should come under offset
> as per the same
> [doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendOffset(java.lang.String,%20java.lang.String)].
> It seems, however, that "UTC" was explicitly removed [when parsed as ZoneText
> for higher Java
> versions|https://github.com/openjdk/jdk/commit/5c3a01591c5c945926636fdc9f164d60b5b4f29e?diff=unified#diff-5fcf976db1c06e8f44a8671356d7e34fdfbf5b057baa852e7c3e015c8797c889R4263]
> As a workaround, we can use {{appendZoneOrOffsetId()}} rather than
> {{appendZoneText().}}
> This ensures the tests are passing for Java 8+ and based on my testing, I
> didn’t see any regression of the change.
> Sample repro code - jdoodle.com/ia/D5e
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26998) Can not find correct root task on compute column statistics for views
[ https://issues.apache.org/jira/browse/HIVE-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682253#comment-17682253 ] Sai Hemanth Gantasala commented on HIVE-26998: -- cc [~krisztiankasa] > Can not find correct root task on compute column statistics for views > - > > Key: HIVE-26998 > URL: https://issues.apache.org/jira/browse/HIVE-26998 > Project: Hive > Issue Type: Bug >Reporter: chiranjeevi >Priority: Minor > > When you run ANALYZE TABLE view_name COMPUTE STATISTICS FOR COLUMNS; > Server should notify > *Expected Result :* > ANALYZE is not supported for views (state=42000,code=10091) like it does for > table stat > *Actual result:* > Error: Error while compiling statement: FAILED: SemanticException Can not > find correct root task -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26850) ColumnInfo of the source table should not be null for the alter view as query
[ https://issues.apache.org/jira/browse/HIVE-26850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17681443#comment-17681443 ] Sai Hemanth Gantasala commented on HIVE-26850: -- Patch merged to master. Closing out this Jira. > ColumnInfo of the source table should not be null for the alter view as query > - > > Key: HIVE-26850 > URL: https://issues.apache.org/jira/browse/HIVE-26850 > Project: Hive > Issue Type: Bug >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The ColumnInfo for the source table of the alter view as query is coming as > null and as a result, the column info is not present for the table. The > ColumnInfo should be non-null. > More details are to be followed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26850) ColumnInfo of the source table should not be null for the alter view as query
[ https://issues.apache.org/jira/browse/HIVE-26850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-26850. -- Fix Version/s: 4.0.0 Resolution: Fixed > ColumnInfo of the source table should not be null for the alter view as query > - > > Key: HIVE-26850 > URL: https://issues.apache.org/jira/browse/HIVE-26850 > Project: Hive > Issue Type: Bug >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The ColumnInfo for the source table of the alter view as query is coming as > null and as a result, the column info is not present for the table. The > ColumnInfo should be non-null. > More details are to be followed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26508) Remove netty transitive dependencies from hcatalog and hbase pom files to avoid CVEs
[ https://issues.apache.org/jira/browse/HIVE-26508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-26508. -- Resolution: Won't Fix > Remove netty transitive dependencies from hcatalog and hbase pom files to > avoid CVEs > > > Key: HIVE-26508 > URL: https://issues.apache.org/jira/browse/HIVE-26508 > Project: Hive > Issue Type: Bug > Components: HBase Handler, HCatalog >Affects Versions: 4.0.0, 4.0.0-alpha-1, 4.0.0-alpha-2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Remove netty transitive dependencies (coming from hadoop related > dependencies) from hcatalog and hbase pom files to avoid CVEs -- This message was sent by Atlassian Jira (v8.20.10#820010)
