[jira] [Created] (HIVE-27782) HS2 crashes with OOM even though there is enough heap space
Pravin Sinha created HIVE-27782: --- Summary: HS2 crashes with OOM even though there is enough heap space Key: HIVE-27782 URL: https://issues.apache.org/jira/browse/HIVE-27782 Project: Hive Issue Type: Bug Reporter: Pravin Sinha Assignee: Pravin Sinha -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27781) HMS crashes with OOM even though there is enough heap space
Pravin Sinha created HIVE-27781: --- Summary: HMS crashes with OOM even though there is enough heap space Key: HIVE-27781 URL: https://issues.apache.org/jira/browse/HIVE-27781 Project: Hive Issue Type: Bug Reporter: Pravin Sinha Assignee: Pravin Sinha -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27765) Backport of HIVE-20052, HIVE-20093, HIVE-20203, HIVE-20290, HIVE-20300, HIVE-20312, HIVE-20044, HIVE-21966
[ https://issues.apache.org/jira/browse/HIVE-27765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-27765: Description: * HIVE-20052: Arrow serde should fill ArrowColumnVector(Decimal) with the given schema precision/scale * HIVE-20093: LlapOutputFomatService: Use ArrowBuf with Netty for Accounting * HIVE-20203: Arrow SerDe leaks a DirectByteBuffer * HIVE-20290: Lazy initialize ArrowColumnarBatchSerDe so it doesn't allocate buffers during GetSplits * HIVE-20300: VectorFileSinkArrowOperator * HIVE-20312: Allow arrow clients to use their own BufferAllocator with LlapOutputFormatService * HIVE-20044: Arrow Serde should pad char values and handle empty strings correctly * HIVE-21966: Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases > Backport of HIVE-20052, HIVE-20093, HIVE-20203, HIVE-20290, HIVE-20300, > HIVE-20312, HIVE-20044, HIVE-21966 > -- > > Key: HIVE-27765 > URL: https://issues.apache.org/jira/browse/HIVE-27765 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.3 >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > > * HIVE-20052: Arrow serde should fill ArrowColumnVector(Decimal) with the > given schema precision/scale > * HIVE-20093: LlapOutputFomatService: Use ArrowBuf with Netty for > Accounting > * HIVE-20203: Arrow SerDe leaks a DirectByteBuffer > * HIVE-20290: Lazy initialize ArrowColumnarBatchSerDe so it doesn't > allocate buffers during GetSplits > * HIVE-20300: VectorFileSinkArrowOperator > * HIVE-20312: Allow arrow clients to use their own BufferAllocator with > LlapOutputFormatService > * HIVE-20044: Arrow Serde should pad char values and handle empty strings > correctly > * HIVE-21966: Llap external client - Arrow Serializer throws > ArrayIndexOutOfBoundsException in some cases -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27765) Backport of HIVE-20052, HIVE-20093, HIVE-20203, HIVE-20290, HIVE-20300, HIVE-20312, HIVE-20044, HIVE-21966
[ https://issues.apache.org/jira/browse/HIVE-27765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan resolved HIVE-27765. - Fix Version/s: 3.2.0 Resolution: Fixed > Backport of HIVE-20052, HIVE-20093, HIVE-20203, HIVE-20290, HIVE-20300, > HIVE-20312, HIVE-20044, HIVE-21966 > -- > > Key: HIVE-27765 > URL: https://issues.apache.org/jira/browse/HIVE-27765 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27765) Backport of HIVE-20052, HIVE-20093, HIVE-20203, HIVE-20290, HIVE-20300, HIVE-20312, HIVE-20044, HIVE-21966
[ https://issues.apache.org/jira/browse/HIVE-27765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-27765: Affects Version/s: 3.1.3 (was: 3.2.0) > Backport of HIVE-20052, HIVE-20093, HIVE-20203, HIVE-20290, HIVE-20300, > HIVE-20312, HIVE-20044, HIVE-21966 > -- > > Key: HIVE-27765 > URL: https://issues.apache.org/jira/browse/HIVE-27765 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.3 >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-22989) Don't close parent classloader when session being closed
[ https://issues.apache.org/jira/browse/HIVE-22989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng reassigned HIVE-22989: -- Assignee: (was: Zhihua Deng) > Don't close parent classloader when session being closed > > > Key: HIVE-22989 > URL: https://issues.apache.org/jira/browse/HIVE-22989 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-22989.patch > > Time Spent: 20m > Remaining Estimate: 0h > > When hiveserver2 loads udfs, Registry will use the session's specified > classloader to load them and cache the classloader. When user don't set the > aux jars, the classloader cached is equal to the session's parent > classloader, in our case, we don't set the aux jars while update the > session's parent classloader periodicity to update user jars dynamically. > It's should do a sanity check when Registry closes the cached classloaders. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-19818) SessionState getQueryId returns an empty string
[ https://issues.apache.org/jira/browse/HIVE-19818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16503185#comment-16503185 ] Zhihua Deng edited comment on HIVE-19818 at 10/10/23 1:24 AM: -- One possible way to solve the problem is that getting the query id from the job configuration : HiveConf.getVar(job, HiveConf.ConfVars.HIVEQUERYID, "").trim() was (Author: dengzh): one possible way to solve the problem is that getting the query id from the job configuration like: HiveConf.getVar(job, HiveConf.ConfVars.HIVEQUERYID, "").trim() > SessionState getQueryId returns an empty string > --- > > Key: HIVE-19818 > URL: https://issues.apache.org/jira/browse/HIVE-19818 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Attachments: HIVE-19818.patch > > > When we execute sql asynchronously, a new configuration based on the session > holds will be created and passed to the driver instance, which resulting to > return an empty string when SessionState#getQueryId called later on. This > problem can be seen in HadoopJobExecHelper.java. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-23720) Background task may not be interrupted when operation being canceled or timeout
[ https://issues.apache.org/jira/browse/HIVE-23720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17139107#comment-17139107 ] Zhihua Deng edited comment on HIVE-23720 at 10/10/23 1:23 AM: -- Need a further research as I see there no releaseDriverContext() is called. https://issues.apache.org/jira/browse/HIVE-16426 was (Author: dengzh): Should do a further research as I see there no releaseDriverContext() is called. https://issues.apache.org/jira/browse/HIVE-16426 > Background task may not be interrupted when operation being canceled or > timeout > --- > > Key: HIVE-23720 > URL: https://issues.apache.org/jira/browse/HIVE-23720 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Now SQLOperation cancels the background task only when the condition is met: > if (shouldRunAsync() && state != OperationState.CANCELED && state != > OperationState.TIMEDOUT) > The condition is evaluated to false when state is OperationState.CANCELED or > OperationState.TIMEDOUT, but operations in such states should stop the > background tasks to release resources. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-23720) Background task may not be interrupted when operation being canceled or timeout
[ https://issues.apache.org/jira/browse/HIVE-23720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140382#comment-17140382 ] Zhihua Deng edited comment on HIVE-23720 at 10/10/23 1:23 AM: -- The running Task can be interrupted by releaseTaskQueue() in driver::close(), it's ok to shutdown by this instead of canceling the background. was (Author: dengzh): The running Task can be interrupted by releaseTaskQueue() in driver::close(), so it's ok to shutdown by this instead of canceling the background. > Background task may not be interrupted when operation being canceled or > timeout > --- > > Key: HIVE-23720 > URL: https://issues.apache.org/jira/browse/HIVE-23720 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Now SQLOperation cancels the background task only when the condition is met: > if (shouldRunAsync() && state != OperationState.CANCELED && state != > OperationState.TIMEDOUT) > The condition is evaluated to false when state is OperationState.CANCELED or > OperationState.TIMEDOUT, but operations in such states should stop the > background tasks to release resources. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-23185) Historic queries lost after HS2 restart
[ https://issues.apache.org/jira/browse/HIVE-23185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng reassigned HIVE-23185: -- Assignee: (was: Zhihua Deng) > Historic queries lost after HS2 restart > --- > > Key: HIVE-23185 > URL: https://issues.apache.org/jira/browse/HIVE-23185 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Zhihua Deng >Priority: Major > > QueryInfoCache caches historic queries in memory, when HS2 restart due to OOM > or upgrade, the queries are no longer seen at webui. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-23727) Improve SQLOperation log handling when canceling background
[ https://issues.apache.org/jira/browse/HIVE-23727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng reassigned HIVE-23727: -- Assignee: (was: Zhihua Deng) > Improve SQLOperation log handling when canceling background > --- > > Key: HIVE-23727 > URL: https://issues.apache.org/jira/browse/HIVE-23727 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 4.5h > Remaining Estimate: 0h > > The SQLOperation checks _if (shouldRunAsync() && state != > OperationState.CANCELED && state != OperationState.TIMEDOUT)_ to cancel the > background task. If true, the state should not be OperationState.CANCELED, so > logging under the state == OperationState.CANCELED should never happen. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-24422) Throw SemanticException when CTE alias is conflicted with table name
[ https://issues.apache.org/jira/browse/HIVE-24422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252750#comment-17252750 ] Zhihua Deng edited comment on HIVE-24422 at 10/10/23 1:21 AM: -- Seems I was wrong. In the trunk, the query {code:java} with game_info as ( select distinct ext_id, dev_app_id, game_name from game_info_extend ) select count(game_name) from game_info;{code} The game_info is materialized, this can reduce recomputing of game_info no matter how many times the game_info is referenced. was (Author: dengzh): Seems I am wrong. In the trunk, the query {code:java} with game_info as ( select distinct ext_id, dev_app_id, game_name from game_info_extend ) select count(game_name) from game_info;{code} The game_info is firstly materialized, this can reduce recomputing of game_info no matter how many times the game_info is referenced , the materialized view is used other than the table game_info for analyzing the tree later on. > Throw SemanticException when CTE alias is conflicted with table name > > > Key: HIVE-24422 > URL: https://issues.apache.org/jira/browse/HIVE-24422 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > If the alias of CTE is conflicted with the table name, we use the alias > fetching the table other than replacing it with the ASTNode tree, this may > cause some confusing problems. For example: > {noformat} > create table game_info (game_name string); > with game_info as ( > select distinct ext_id, dev_app_id, game_name > from game_info_extend ) > select count(game_name) from game_info;{noformat} > The query will return the number of rows of the table game_info, instead of > the game_info_extend. Maybe we should better throw an exception to avoid such > cases. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-24351) Report progress to prevent merge task from timeout
[ https://issues.apache.org/jira/browse/HIVE-24351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng reassigned HIVE-24351: -- Assignee: (was: Zhihua Deng) > Report progress to prevent merge task from timeout > -- > > Key: HIVE-24351 > URL: https://issues.apache.org/jira/browse/HIVE-24351 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > If the MergeFileTask tries to merge lots of empty files, the task may be > terminated due to task timeout. It’s rare, but it happens. Report the > progress regularly to prevent the mapper from timeout. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-24639) Raises SemanticException other than ClassCastException when filter has non-boolean expressions
[ https://issues.apache.org/jira/browse/HIVE-24639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng reassigned HIVE-24639: -- Assignee: (was: Zhihua Deng) > Raises SemanticException other than ClassCastException when filter has > non-boolean expressions > -- > > Key: HIVE-24639 > URL: https://issues.apache.org/jira/browse/HIVE-24639 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 50m > Remaining Estimate: 0h > > Sometimes we see ClassCastException in filters when fetching some rows of a > table or executing the query. The > GenericUDFOPOr/GenericUDFOPAnd/FilterOperator assume that the output of their > conditions should be a boolean, but there is no garanteed. For example: > _select * from ccn_table where src + 1;_ > will throw ClassCastException: > {code:java} > Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to > java.lang.Boolean > at > org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:125) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888) > at > org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:173) > at > org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:153) > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:553) > ...{code} > We'd better to validate the filter during analyzing instead of at runtime and > bring more meaningful messages. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-25294) Optimise the metadata count queries for local mode
[ https://issues.apache.org/jira/browse/HIVE-25294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng reassigned HIVE-25294: -- Assignee: (was: Zhihua Deng) > Optimise the metadata count queries for local mode > -- > > Key: HIVE-25294 > URL: https://issues.apache.org/jira/browse/HIVE-25294 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > When Metastore is in local mode, the client uses his own private HMSHandler > to get the meta data, the HMSHandler should be initialized before being > ready to serve. When the metrics is enabled, HMSHandler will count the number > of db, table, partitions on initializing, which cloud lead to some > performance problems. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27201) Inconsistency between session Hive and thread-local Hive may cause HS2 deadlock
[ https://issues.apache.org/jira/browse/HIVE-27201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17716189#comment-17716189 ] Zhihua Deng edited comment on HIVE-27201 at 10/10/23 1:14 AM: -- Attach a document to explain the case. Thank you all for the test, review and merge the PR! was (Author: dengzh): Attach a document to elaborate the case. Thank you all for the test, review and merge the PR! > Inconsistency between session Hive and thread-local Hive may cause HS2 > deadlock > --- > > Key: HIVE-27201 > URL: https://issues.apache.org/jira/browse/HIVE-27201 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-beta-1 > > Attachments: HIVE-27201.pdf > > Time Spent: 7h 20m > Remaining Estimate: 0h > > The HiveServer2’s server handler can switch to process the operation from > other session, in such case, the Hive cached in ThreadLocal is not the same > as the Hive in SessionState, and can be referenced by another session. > If the two handlers swap their sessions to process the DatabaseMetaData > request, and the HiveMetastoreClientFactory obtains the Hive via Hive.get(), > then there is a chance that the deadlock can happen. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27352) Support both LDAP and Kerberos Auth in HS2
[ https://issues.apache.org/jira/browse/HIVE-27352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729681#comment-17729681 ] Zhihua Deng edited comment on HIVE-27352 at 10/10/23 1:13 AM: -- Thanks for the review and merge, [~hemanth619]! was (Author: dengzh): Thanks for the review and merge, [~hemanth619]! For Http mode, we must configure the new auth method to the end of original hive.server2.authentication in order to keep compatible with the old HS2 client. > Support both LDAP and Kerberos Auth in HS2 > -- > > Key: HIVE-27352 > URL: https://issues.apache.org/jira/browse/HIVE-27352 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-beta-1 > > > Currently, HS2 supports a single form of auth in binary mode, and some > limited multiple auth methods in http mode. Some analysis tools based on Hive > JDBC provide support for mixing both Kerberos and LDAP, however HS2 doesn't > make itself heard for this auth type, both in http and binary mode. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27597) Implement JDBC Connector for HiveServer
[ https://issues.apache.org/jira/browse/HIVE-27597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773426#comment-17773426 ] zhangbutao commented on HIVE-27597: --- [~ngangam] I probably still don't have edit permission, and i can't find the edit button. Could you please check it again? Thanks. [https://cwiki.apache.org/confluence/display/Hive/Data+Connectors+in+Hive] > Implement JDBC Connector for HiveServer > > > Key: HIVE-27597 > URL: https://issues.apache.org/jira/browse/HIVE-27597 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Labels: pull-request-available > > The initial idea of having a thrift based connector, that would enable Hive > Metastore to use thrift APIs to interact with another metastore from another > cluster, has some limitations. Features like column masking support become a > challenge as we may bypass the authz controls on the remote cluster. > Instead if we could federate a query from one instance of HS2 to another > instance of HS2 over JDBC, we would address the above concerns. This will > atleast give us the ability to access tables across cluster boundaries. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27597) Implement JDBC Connector for HiveServer
[ https://issues.apache.org/jira/browse/HIVE-27597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam resolved HIVE-27597. -- Resolution: Fixed Fix has been merged to master. Thank you [~ayushtkn] and [~zhangbutao] for the review. Closing the jira. [~zhangbutao] I have granted you write access to the Data connectors page. Hope it works. Thank you > Implement JDBC Connector for HiveServer > > > Key: HIVE-27597 > URL: https://issues.apache.org/jira/browse/HIVE-27597 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Labels: pull-request-available > > The initial idea of having a thrift based connector, that would enable Hive > Metastore to use thrift APIs to interact with another metastore from another > cluster, has some limitations. Features like column masking support become a > challenge as we may bypass the authz controls on the remote cluster. > Instead if we could federate a query from one instance of HS2 to another > instance of HS2 over JDBC, we would address the above concerns. This will > atleast give us the ability to access tables across cluster boundaries. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27780) Implement direct SQL for get_all_functions
[ https://issues.apache.org/jira/browse/HIVE-27780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27780: -- Labels: pull-request-available (was: ) > Implement direct SQL for get_all_functions > -- > > Key: HIVE-27780 > URL: https://issues.apache.org/jira/browse/HIVE-27780 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: zhangbutao >Assignee: zhangbutao >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27771) Iceberg: Allow expire snapshot by time range
[ https://issues.apache.org/jira/browse/HIVE-27771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-27771. - Fix Version/s: 4.0.0 Resolution: Fixed > Iceberg: Allow expire snapshot by time range > > > Key: HIVE-27771 > URL: https://issues.apache.org/jira/browse/HIVE-27771 > Project: Hive > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Allow expiring snapshot by time range. > Alter table ice01 execute expire_snapshot BETWEEN 'some time' AND 'some time' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27771) Iceberg: Allow expire snapshot by time range
[ https://issues.apache.org/jira/browse/HIVE-27771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773254#comment-17773254 ] Ayush Saxena commented on HIVE-27771: - Committed to master. Thanx [~dkuzmenko] for the review!!! > Iceberg: Allow expire snapshot by time range > > > Key: HIVE-27771 > URL: https://issues.apache.org/jira/browse/HIVE-27771 > Project: Hive > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > > Allow expiring snapshot by time range. > Alter table ice01 execute expire_snapshot BETWEEN 'some time' AND 'some time' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27780) Implement direct SQL for get_all_functions
[ https://issues.apache.org/jira/browse/HIVE-27780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangbutao reassigned HIVE-27780: - Assignee: zhangbutao > Implement direct SQL for get_all_functions > -- > > Key: HIVE-27780 > URL: https://issues.apache.org/jira/browse/HIVE-27780 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: zhangbutao >Assignee: zhangbutao >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27780) Implement direct SQL for get_all_functions
zhangbutao created HIVE-27780: - Summary: Implement direct SQL for get_all_functions Key: HIVE-27780 URL: https://issues.apache.org/jira/browse/HIVE-27780 Project: Hive Issue Type: Improvement Components: Standalone Metastore Reporter: zhangbutao -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27779) Iceberg: Drop partition support
[ https://issues.apache.org/jira/browse/HIVE-27779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27779: -- Labels: pull-request-available (was: ) > Iceberg: Drop partition support > --- > > Key: HIVE-27779 > URL: https://issues.apache.org/jira/browse/HIVE-27779 > Project: Hive > Issue Type: Improvement >Reporter: Sourabh Badhya >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > > A logical extension of TRUNCATE PARTITION however, DROP PARTITION allows > expressions with >, <, >=, <=, != etc. > The syntax is something as follows - > {code:java} > alter table tableName drop partition (c='US', d<'2');{code} > Drop partition command also allows multiple partition expressions as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27777) CBO fails on multi insert overwrites with common group expression
[ https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis updated HIVE-2: --- Description: The following statement is failing in CBO: {code:java} FROM (select key, f1 FROM tbl1 where key=5) a INSERT OVERWRITE TABLE tbl2 partition(key=5) select f1 WHERE key > 0 GROUP by f1 INSERT OVERWRITE TABLE tbl2 partition(key=6) select f1 WHERE key > 0 GROUP by f1; {code} The failure happens when there is a filter to a constant value in the FROM clause ,the value is referenced in the filter in the INSERT OVERWRITE, and there is a common group existing across the insert overwrites. CBO is pulling up the key = 5 expression into the select clause as a constant (i.e. select 5 key, f1 FROM tbl1 where key = 5). After it gets converted back into AST and then re-compiled, there is code in the common group method that expects all columns to be non-constants which is causing the failure. The failure stacktrace is shown below: {noformat} org.apache.hadoop.hive.ql.parse.SemanticException: Line 6:16 Expression not in GROUP BY key '0' at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:13509) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:13451) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:13419) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:3727) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:3707) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlan1ReduceMultiGBY(SemanticAnalyzer.java:6514) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:11415) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12343) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12209) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:634) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13073) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:465) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:180) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224) at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:107) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:519) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:471) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:436) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:430) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:257) at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:425) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:356) at org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:733) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:703) at org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:115) at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157) {noformat} was: The following statement is failing in CBO: {code:java} FROM (select key, f1 FROM tbl1 where key=5) a INSERT OVERWRITE TABLE tbl2 partition(key=5) select f1 WHERE key > 0 GROUP by f1 INSERT OVERWRITE TABLE tbl2 partition(key=6) select f1 WHERE key > 0 GROUP by f1; {code} The failure happens when there is a filter to a constant value in the FROM clause ,the value is referenced in the filter in the INSERT OVERWRITE, and there is a common group existing across the insert overwrites. CBO is pulling up the key = 5 expression into the select clause as a constant (i.e. select 5 key, f1 FROM tbl1 where key = 5). After it gets converted back into AST and then re-compiled, there is code in the common group method that expects all columns to be non-constants which is causing the failiure. > CBO fails on multi insert overwrites with com
[jira] [Created] (HIVE-27779) Iceberg: Drop partition support
Sourabh Badhya created HIVE-27779: - Summary: Iceberg: Drop partition support Key: HIVE-27779 URL: https://issues.apache.org/jira/browse/HIVE-27779 Project: Hive Issue Type: Improvement Reporter: Sourabh Badhya Assignee: Sourabh Badhya A logical extension of TRUNCATE PARTITION however, DROP PARTITION allows expressions with >, <, >=, <=, != etc. The syntax is something as follows - {code:java} alter table tableName drop partition (c='US', d<'2');{code} Drop partition command also allows multiple partition expressions as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27755) Quote identifiers in SQL emitted by SchemaTool for MySQL
[ https://issues.apache.org/jira/browse/HIVE-27755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773173#comment-17773173 ] Stamatis Zampetakis commented on HIVE-27755: The queries just above do not come from the validate command but from the test itself. The {{TestSchemaToolForMetastore.testValidateSequences}} case sends some [hardcoded DML commands|https://github.com/apache/hive/blob/20be17d19c19a1482dc3b1b753ce1f342b940e36/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/tools/schematool/TestSchemaToolForMetastore.java#L122] before/after validation. > Quote identifiers in SQL emitted by SchemaTool for MySQL > > > Key: HIVE-27755 > URL: https://issues.apache.org/jira/browse/HIVE-27755 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Affects Versions: 4.0.0-beta-1 >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Attachments: TestMysql-upgrade-after.txt, > TestMysql-upgrade-before.txt, > TestSchemaToolForMetastore-validateSequences-after.txt, > TestSchemaToolForMetastore-validateSequences-before.txt, > TestSchemaToolForMetastore-validateTables-after.txt, > TestSchemaToolForMetastore-validateTables-before.txt > > > Various SchemaTool options/tasks (e.g., "validate") generate and run SQL > statements on the underlying database. Depending on the database identifiers > in the SQL statements may be quoted (see > [https://github.com/apache/hive/blob/2dbfbeefc1a73d6a50f1c829658846fc827fc780/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/HiveSchemaHelper.java#L173]). > Currently, all identifiers are quoted when the database is Postgres and this > tickets aims to do the same for MySQL/MariaDB. > The main motivation behind this change is to avoid unexpected surprises and > query failures when/if the database decides to turn some of the > tables/columns we are using internally to reserved keywords. > As a concrete example, the Percona fork of MySQL recently turned > SEQUENCE_TABLE into a reserved keyword > ([https://docs.percona.com/percona-server/8.0/flexibility/sequence_table.html]) > and this comes in conflict with our internal metastore table. > The installation scripts do not fail since in that case SEQUENCE_TABLE is > quoted > ([https://github.com/apache/hive/blob/2dbfbeefc1a73d6a50f1c829658846fc827fc780/standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0-beta-2.mysql.sql#L447]) > but validation queries emitted by the SchemaTool will fail > ([https://github.com/apache/hive/blob/2dbfbeefc1a73d6a50f1c829658846fc827fc780/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskValidate.java#L117]) > if we don't use quoted identifiers. -- This message was sent by Atlassian Jira (v8.20.10#820010)