[jira] [Updated] (HIVE-27494) Deduplicate the task result that generated by more branches in union all
[ https://issues.apache.org/jira/browse/HIVE-27494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-27494: --- Description: HIVE-23891 adds the ability to deduplicate the task result that under the directory, //_tmp.-ext-1//HIVE_UNION_SUBDIR_1, but turns out to ignore taking the same action to the output directory for the same query: //_tmp.-ext-1//HIVE_UNION_SUBDIR_2. So user may still have the same data duplication problem upon multiple tez task attempts. was: HIVE-23891 adds the ability to deduplicate the task result that under the directory, //_tmp.-ext-1//HIVE_UNION_SUBDIR_1, but turns out to ignore taking the same action to the directory for the same query: //_tmp.-ext-1//HIVE_UNION_SUBDIR_2. So user may still have the same data duplication problem in multiple tez task attempts. > Deduplicate the task result that generated by more branches in union all > > > Key: HIVE-27494 > URL: https://issues.apache.org/jira/browse/HIVE-27494 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-beta-1 > > Attachments: ddl.q, explain.output > > > HIVE-23891 adds the ability to deduplicate the task result that under the > directory, > //_tmp.-ext-1//HIVE_UNION_SUBDIR_1, > but turns out to ignore taking the same action to the output directory for > the same query: > //_tmp.-ext-1//HIVE_UNION_SUBDIR_2. > So user may still have the same data duplication problem upon multiple tez > task attempts. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28019) Fix query type information in proto files for load queries
[ https://issues.apache.org/jira/browse/HIVE-28019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-28019: Summary: Fix query type information in proto files for load queries (was: Fix query type information in proto files for load and explain queries) > Fix query type information in proto files for load queries > -- > > Key: HIVE-28019 > URL: https://issues.apache.org/jira/browse/HIVE-28019 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > > Certain query types like LOAD, export, import and explain queries did not > produce the right Hive operation type -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28207) NullPointerException is thrown when checking column uniqueness
[
https://issues.apache.org/jira/browse/HIVE-28207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shohei Okumiya updated HIVE-28207:
--
Description:
In some cases, we skip checking null. For example, the last statement in the
following set of queries fails with NPE.
{code:java}
CREATE TABLE `store_sales` (`ss_item_sk` bigint);
CREATE TABLE `household_demographics` (`hd_demo_sk` bigint);
CREATE TABLE `item` (`i_item_sk` bigint);
ALTER TABLE `store_sales` ADD CONSTRAINT `pk_ss` PRIMARY KEY (`ss_item_sk`)
DISABLE NOVALIDATE RELY;
ALTER TABLE `item` ADD CONSTRAINT `pk_i` PRIMARY KEY (`i_item_sk`) DISABLE
NOVALIDATE RELY;
ALTER TABLE `store_sales` ADD CONSTRAINT `ss_i` FOREIGN KEY (`ss_item_sk`)
REFERENCES `item`(`i_item_sk`) DISABLE NOVALIDATE RELY;
EXPLAIN
SELECT i_item_sk
FROM store_sales, household_demographics, item
WHERE ss_item_sk = i_item_sk{code}
The NPE happens with HiveJoinConstraintsRule in the above case.
{code:java}
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement:
FAILED: NullPointerException null
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:376)
~[hive-service-4.0.0.jar:4.0.0]
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:214)
~[hive-service-4.0.0.jar:4.0.0]
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:270)
~[hive-service-4.0.0.jar:4.0.0]
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:286)
~[hive-service-4.0.0.jar:4.0.0]
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:557)
~[hive-service-4.0.0.jar:4.0.0]
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:542)
~[hive-service-4.0.0.jar:4.0.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_275]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_275]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_275]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
~[hive-service-4.0.0.jar:4.0.0]
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
~[hive-service-4.0.0.jar:4.0.0]
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
~[hive-service-4.0.0.jar:4.0.0]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_275]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_275]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
~[hadoop-common-3.3.6.jar:?]
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
~[hive-service-4.0.0.jar:4.0.0]
at com.sun.proxy.$Proxy42.executeStatementAsync(Unknown Source) ~[?:?]
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:316)
~[hive-service-4.0.0.jar:4.0.0]
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:652)
~[hive-service-4.0.0.jar:4.0.0]
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1670)
~[hive-exec-4.0.0.jar:4.0.0]
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1650)
~[hive-exec-4.0.0.jar:4.0.0]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
~[hive-exec-4.0.0.jar:4.0.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
~[hive-exec-4.0.0.jar:4.0.0]
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
~[hive-service-4.0.0.jar:4.0.0]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
~[hive-exec-4.0.0.jar:4.0.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinConstraintsRule.onMatch(HiveJoinConstraintsRule.java:277)
~[hive-exec-4.0.0.jar:4.0.0]
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
~[hive-exec-4.0.0.jar:4.0.0]
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
~[hive-exec-4.0.0.jar:4.0.0]
at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
~[hive-exec-4.0.0.jar:4.0.0]
at
org.apache
[jira] [Commented] (HIVE-28207) NullPointerException is thrown when checking column uniqueness
[
https://issues.apache.org/jira/browse/HIVE-28207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17839600#comment-17839600
]
Shohei Okumiya commented on HIVE-28207:
---
Thanks. I attached a stack trace to the description.
> NullPointerException is thrown when checking column uniqueness
> --
>
> Key: HIVE-28207
> URL: https://issues.apache.org/jira/browse/HIVE-28207
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0
>Reporter: Shohei Okumiya
>Assignee: Shohei Okumiya
>Priority: Major
> Labels: pull-request-available
>
> In some cases, we skip checking null. For example, the last statement in the
> following set of queries fails with NPE.
> {code:java}
> CREATE TABLE `store_sales` (`ss_item_sk` bigint);
> CREATE TABLE `household_demographics` (`hd_demo_sk` bigint);
> CREATE TABLE `item` (`i_item_sk` bigint);
> ALTER TABLE `store_sales` ADD CONSTRAINT `pk_ss` PRIMARY KEY (`ss_item_sk`)
> DISABLE NOVALIDATE RELY;
> ALTER TABLE `item` ADD CONSTRAINT `pk_i` PRIMARY KEY (`i_item_sk`) DISABLE
> NOVALIDATE RELY;
> ALTER TABLE `store_sales` ADD CONSTRAINT `ss_i` FOREIGN KEY (`ss_item_sk`)
> REFERENCES `item`(`i_item_sk`) DISABLE NOVALIDATE RELY;
> EXPLAIN
> SELECT i_item_sk
> FROM store_sales, household_demographics, item
> WHERE ss_item_sk = i_item_sk{code}
> The NPE happens with HiveJoinConstraintsRule in the above case.
> {code:java}
> org.apache.hive.service.cli.HiveSQLException: Error while compiling
> statement: FAILED: NullPointerException null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:376)
> ~[hive-service-4.0.0.jar:4.0.0]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:214)
> ~[hive-service-4.0.0.jar:4.0.0]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:270)
> ~[hive-service-4.0.0.jar:4.0.0]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:286)
> ~[hive-service-4.0.0.jar:4.0.0]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:557)
> ~[hive-service-4.0.0.jar:4.0.0]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:542)
> ~[hive-service-4.0.0.jar:4.0.0]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_275]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_275]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_275]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> ~[hive-service-4.0.0.jar:4.0.0]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-4.0.0.jar:4.0.0]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-4.0.0.jar:4.0.0]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_275]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_275]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> ~[hadoop-common-3.3.6.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-4.0.0.jar:4.0.0]
> at com.sun.proxy.$Proxy42.executeStatementAsync(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:316)
> ~[hive-service-4.0.0.jar:4.0.0]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:652)
> ~[hive-service-4.0.0.jar:4.0.0]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1670)
> ~[hive-exec-4.0.0.jar:4.0.0]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1650)
> ~[hive-exec-4.0.0.jar:4.0.0]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> ~[hive-exec-4.0.0.jar:4.0.0]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> ~[hive-exec-4.0.0.jar:4.0.0]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> ~[hive-service-4.0.0.jar:4.0.0]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
> ~[hive-exec-4.0.0.jar:4.0.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWo
[jira] [Commented] (HIVE-28207) NullPointerException is thrown when checking column uniqueness
[
https://issues.apache.org/jira/browse/HIVE-28207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17839593#comment-17839593
]
Stamatis Zampetakis commented on HIVE-28207:
Thanks for reporting this issue [~okumin] ! When queries fail with exceptions
it is a good practice to include the stacktrace in the description (inside
\{noformat} tags) of the Jira ticket since it helps in the understanding of the
problem and is also very helpful to people who may encounter the same or
similar problem to identify the correct bug report via JIRA/Google search.
Please include it if possible.
> NullPointerException is thrown when checking column uniqueness
> --
>
> Key: HIVE-28207
> URL: https://issues.apache.org/jira/browse/HIVE-28207
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0
>Reporter: Shohei Okumiya
>Assignee: Shohei Okumiya
>Priority: Major
> Labels: pull-request-available
>
> In some cases, we skip checking null. For example, the last statement in the
> following set of queries fails with NPE.
> {code:java}
> CREATE TABLE `store_sales` (`ss_item_sk` bigint);
> CREATE TABLE `household_demographics` (`hd_demo_sk` bigint);
> CREATE TABLE `item` (`i_item_sk` bigint);
> ALTER TABLE `store_sales` ADD CONSTRAINT `pk_ss` PRIMARY KEY (`ss_item_sk`)
> DISABLE NOVALIDATE RELY;
> ALTER TABLE `item` ADD CONSTRAINT `pk_i` PRIMARY KEY (`i_item_sk`) DISABLE
> NOVALIDATE RELY;
> ALTER TABLE `store_sales` ADD CONSTRAINT `ss_i` FOREIGN KEY (`ss_item_sk`)
> REFERENCES `item`(`i_item_sk`) DISABLE NOVALIDATE RELY;
> EXPLAIN
> SELECT i_item_sk
> FROM store_sales, household_demographics, item
> WHERE ss_item_sk = i_item_sk {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28188) Upgrade PostGres to 42.5.6
[ https://issues.apache.org/jira/browse/HIVE-28188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaspati Krishnatri updated HIVE-28188: Attachment: mvn_dependency_tree.txt > Upgrade PostGres to 42.5.6 > -- > > Key: HIVE-28188 > URL: https://issues.apache.org/jira/browse/HIVE-28188 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: Security, pull-request-available > Attachments: mvn_dependency_tree.txt > > > Upgrade Postgres to 42.5.6 to target critical CVEs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28190) Materialized view rebuild lock heart-beating is broken
[
https://issues.apache.org/jira/browse/HIVE-28190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17839529#comment-17839529
]
Denys Kuzmenko commented on HIVE-28190:
---
[~InvisibleProgrammer], could you please cherry-pick this into branch-4.0?
> Materialized view rebuild lock heart-beating is broken
> --
>
> Key: HIVE-28190
> URL: https://issues.apache.org/jira/browse/HIVE-28190
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Zsolt Miskolczi
>Assignee: Zsolt Miskolczi
>Priority: Critical
> Labels: hive-4.0.1-must, pull-request-available
> Fix For: 4.1.0
>
>
> It fails with the following error:
> {code:java}
> org.springframework.dao.InvalidDataAccessApiUsageException: SQL [UPDATE
> "MATERIALIZATION_REBUILD_LOCKS" SET "MRL_LAST_HEARTBEAT" = 1712571919559
> WHERE "MRL_TXN_ID" = 2297 AND "MRL_DB_NAME" = ? AND "MRL_TBL_NAME" = ?]:
> given 2 parameters but expected 0 {code}
> We didn't spot it so far as when the heartbeat of materialized view fails
> with an error, it doesn't affect the rebuild query run. So that it can be
> only spotted by actively watching the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-28190) Materialized view rebuild lock heart-beating is broken
[
https://issues.apache.org/jira/browse/HIVE-28190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zsolt Miskolczi resolved HIVE-28190.
Fix Version/s: 4.1.0
Resolution: Done
> Materialized view rebuild lock heart-beating is broken
> --
>
> Key: HIVE-28190
> URL: https://issues.apache.org/jira/browse/HIVE-28190
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Zsolt Miskolczi
>Assignee: Zsolt Miskolczi
>Priority: Critical
> Labels: hive-4.0.1-must, pull-request-available
> Fix For: 4.1.0
>
>
> It fails with the following error:
> {code:java}
> org.springframework.dao.InvalidDataAccessApiUsageException: SQL [UPDATE
> "MATERIALIZATION_REBUILD_LOCKS" SET "MRL_LAST_HEARTBEAT" = 1712571919559
> WHERE "MRL_TXN_ID" = 2297 AND "MRL_DB_NAME" = ? AND "MRL_TBL_NAME" = ?]:
> given 2 parameters but expected 0 {code}
> We didn't spot it so far as when the heartbeat of materialized view fails
> with an error, it doesn't affect the rebuild query run. So that it can be
> only spotted by actively watching the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-28190) Materialized view rebuild lock heart-beating is broken
[
https://issues.apache.org/jira/browse/HIVE-28190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17839527#comment-17839527
]
Denys Kuzmenko commented on HIVE-28190:
---
Merged to master.
Thanks for the fix [~InvisibleProgrammer] and [~kkasa] for the review!
> Materialized view rebuild lock heart-beating is broken
> --
>
> Key: HIVE-28190
> URL: https://issues.apache.org/jira/browse/HIVE-28190
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Zsolt Miskolczi
>Assignee: Zsolt Miskolczi
>Priority: Critical
> Labels: hive-4.0.1-must, pull-request-available
>
> It fails with the following error:
> {code:java}
> org.springframework.dao.InvalidDataAccessApiUsageException: SQL [UPDATE
> "MATERIALIZATION_REBUILD_LOCKS" SET "MRL_LAST_HEARTBEAT" = 1712571919559
> WHERE "MRL_TXN_ID" = 2297 AND "MRL_DB_NAME" = ? AND "MRL_TBL_NAME" = ?]:
> given 2 parameters but expected 0 {code}
> We didn't spot it so far as when the heartbeat of materialized view fails
> with an error, it doesn't affect the rebuild query run. So that it can be
> only spotted by actively watching the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27102) Upgrade Calcite to 1.33.0 and Avatica to 1.23.0
[ https://issues.apache.org/jira/browse/HIVE-27102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17839518#comment-17839518 ] Stamatis Zampetakis commented on HIVE-27102: Hey [~frankgrimes97] , Calcite upgrades are rather complex but we will try to advance this work in the next few weeks and hopefully have it in 4.1.0. Other than that its worth mentioning that CVE-2020-13955, and CVE-2022-39135 are probably not exploitable via Hive since the respective codepath does not seem to be used. > Upgrade Calcite to 1.33.0 and Avatica to 1.23.0 > --- > > Key: HIVE-27102 > URL: https://issues.apache.org/jira/browse/HIVE-27102 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > > New versions for Calcite and Avatica are available so we should upgrade to > them. > I had some WIP in HIVE-26610 for upgrading calcite to 1.32.0 but given that > the work was not in very advanced state it is preferred to jump directly to > 1.33.0. > Avatica must be inline with Calcite so both need to be updated at the same > time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
