[jira] [Commented] (HIVE-28036) Move checkLock method to TestTxnDbUtil
[ https://issues.apache.org/jira/browse/HIVE-28036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17835578#comment-17835578 ] László Bodor commented on HIVE-28036: - merged to master, thanks [~dkuzmenko] and [~InvisibleProgrammer] for the reviews! > Move checkLock method to TestTxnDbUtil > -- > > Key: HIVE-28036 > URL: https://issues.apache.org/jira/browse/HIVE-28036 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > > Referring to a method from a different unit test class is crazy, we need to > move it. > When calling a method like TestDbTxnManager2.checkLock, the @ClassRule of > TestDbTxnManager2 is called which can potentially screw up things, so this is > kind of blocking some stuff in HIVE-27972 as well. Unit test classes should > only depend on each other in the same inheritance chain. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-28036) Move checkLock method to TestTxnDbUtil
[ https://issues.apache.org/jira/browse/HIVE-28036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor resolved HIVE-28036. - Resolution: Fixed > Move checkLock method to TestTxnDbUtil > -- > > Key: HIVE-28036 > URL: https://issues.apache.org/jira/browse/HIVE-28036 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > > Referring to a method from a different unit test class is crazy, we need to > move it. > When calling a method like TestDbTxnManager2.checkLock, the @ClassRule of > TestDbTxnManager2 is called which can potentially screw up things, so this is > kind of blocking some stuff in HIVE-27972 as well. Unit test classes should > only depend on each other in the same inheritance chain. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28193) CBO fallback can overwrite the real error message when EXCEPT or INTERSECT are used
[
https://issues.apache.org/jira/browse/HIVE-28193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-28193:
--
Labels: pull-request-available (was: )
> CBO fallback can overwrite the real error message when EXCEPT or INTERSECT
> are used
> ---
>
> Key: HIVE-28193
> URL: https://issues.apache.org/jira/browse/HIVE-28193
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 4.0.0
>Reporter: Shohei Okumiya
>Assignee: Shohei Okumiya
>Priority: Major
> Labels: pull-request-available
>
> With `hive.cbo.fallback.strategy=CONSERVATIVE`, Hive retries to compile an
> SQL without CBO when a compile error happens. We sometimes see an unkind
> error message when EXCEPT or INTERSECT are used. The fallback non-CBO mode
> doesn't support those operators, so Hive asked us to enable CBO.
> {code:java}
> 0: jdbc:hive2://hive-hiveserver2:1/defaul> select cast(0 as bigint) = '1'
> EXCEPT select cast(1 as bigint) = '1';
> Error: Error while compiling statement: FAILED: SemanticException EXCEPT and
> INTERSECT operations are only supported with Cost Based Optimizations
> enabled. Please set 'hive.cbo.enable' to true! (state=42000,code=4)
> 0: jdbc:hive2://hive-hiveserver2:1/defaul> set
> hive.cbo.fallback.strategy=NEVER;
> No rows affected (0.035 seconds)
> 0: jdbc:hive2://hive-hiveserver2:1/defaul> select cast(0 as bigint) = '1'
> EXCEPT select cast(1 as bigint) = '1';
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Wrong arguments ''1'': Unsafe compares between different types are disabled
> for safety reasons. If you know what you are doing, please set
> hive.strict.checks.type.safety to false and make sure that hive.mapred.mode
> is not set to 'strict' to proceed. Note that you may get errors or incorrect
> results if you make a mistake while using some of the unsafe features.
> (state=42000,code=4) {code}
> It is inconvenient, especially when the CBO mode fails because of
> `hive.strict.checks.*`, because we can't find the true cause of an error
> message.
> We plan to disable the fallback behavior or deprecate the non-CBO mode.
> However, we sometimes need to rely on the features at this point.
> * https://issues.apache.org/jira/browse/HIVE-27831
> * https://issues.apache.org/jira/browse/HIVE-27830
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28193) CBO fallback can overwrite the real error message when EXCEPT or INTERSECT are used
Shohei Okumiya created HIVE-28193:
-
Summary: CBO fallback can overwrite the real error message when
EXCEPT or INTERSECT are used
Key: HIVE-28193
URL: https://issues.apache.org/jira/browse/HIVE-28193
Project: Hive
Issue Type: Improvement
Components: CBO
Affects Versions: 4.0.0
Reporter: Shohei Okumiya
Assignee: Shohei Okumiya
With `hive.cbo.fallback.strategy=CONSERVATIVE`, Hive retries to compile an SQL
without CBO when a compile error happens. We sometimes see an unkind error
message when EXCEPT or INTERSECT are used. The fallback non-CBO mode doesn't
support those operators, so Hive asked us to enable CBO.
{code:java}
0: jdbc:hive2://hive-hiveserver2:1/defaul> select cast(0 as bigint) = '1'
EXCEPT select cast(1 as bigint) = '1';
Error: Error while compiling statement: FAILED: SemanticException EXCEPT and
INTERSECT operations are only supported with Cost Based Optimizations enabled.
Please set 'hive.cbo.enable' to true! (state=42000,code=4)
0: jdbc:hive2://hive-hiveserver2:1/defaul> set
hive.cbo.fallback.strategy=NEVER;
No rows affected (0.035 seconds)
0: jdbc:hive2://hive-hiveserver2:1/defaul> select cast(0 as bigint) = '1'
EXCEPT select cast(1 as bigint) = '1';
Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
Wrong arguments ''1'': Unsafe compares between different types are disabled for
safety reasons. If you know what you are doing, please set
hive.strict.checks.type.safety to false and make sure that hive.mapred.mode is
not set to 'strict' to proceed. Note that you may get errors or incorrect
results if you make a mistake while using some of the unsafe features.
(state=42000,code=4) {code}
It is inconvenient, especially when the CBO mode fails because of
`hive.strict.checks.*`, because we can't find the true cause of an error
message.
We plan to disable the fallback behavior or deprecate the non-CBO mode.
However, we sometimes need to rely on the features at this point.
* https://issues.apache.org/jira/browse/HIVE-27831
* https://issues.apache.org/jira/browse/HIVE-27830
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28192) Iceberg: Fix thread safety issue with PositionDeleteInfo in IOContext
[ https://issues.apache.org/jira/browse/HIVE-28192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-28192: -- Labels: pull-request-available (was: ) > Iceberg: Fix thread safety issue with PositionDeleteInfo in IOContext > - > > Key: HIVE-28192 > URL: https://issues.apache.org/jira/browse/HIVE-28192 > Project: Hive > Issue Type: Task >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28192) Iceberg: Fix thread safety issue with PositionDeleteInfo in IOContext
Simhadri Govindappa created HIVE-28192: -- Summary: Iceberg: Fix thread safety issue with PositionDeleteInfo in IOContext Key: HIVE-28192 URL: https://issues.apache.org/jira/browse/HIVE-28192 Project: Hive Issue Type: Task Reporter: Simhadri Govindappa Assignee: Simhadri Govindappa -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28191) Upgrade Hadoop Version to 3.4.0
[ https://issues.apache.org/jira/browse/HIVE-28191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-28191: -- Labels: pull-request-available (was: ) > Upgrade Hadoop Version to 3.4.0 > --- > > Key: HIVE-28191 > URL: https://issues.apache.org/jira/browse/HIVE-28191 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Affects Versions: 4.0.0-beta-1 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > > hadoop 3.4.0 has been released, this jira will try to upgrade the hadoop > version to 3.4.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-28191) Upgrade Hadoop Version to 3.4.0
[ https://issues.apache.org/jira/browse/HIVE-28191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-28191 started by Shilun Fan. - > Upgrade Hadoop Version to 3.4.0 > --- > > Key: HIVE-28191 > URL: https://issues.apache.org/jira/browse/HIVE-28191 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Affects Versions: 4.0.0-beta-1 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > > hadoop 3.4.0 has been released, this jira will try to upgrade the hadoop > version to 3.4.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28191) Upgrade Hadoop Version to 3.4.0
Shilun Fan created HIVE-28191: - Summary: Upgrade Hadoop Version to 3.4.0 Key: HIVE-28191 URL: https://issues.apache.org/jira/browse/HIVE-28191 Project: Hive Issue Type: Improvement Components: Build Infrastructure Affects Versions: 4.0.0-beta-1 Reporter: Shilun Fan Assignee: Shilun Fan hadoop 3.4.0 has been released, this jira will try to upgrade the hadoop version to 3.4.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28190) Fix breaking bug in heartbeatLockMaterializationRebuild
[
https://issues.apache.org/jira/browse/HIVE-28190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-28190:
--
Labels: pull-request-available (was: )
> Fix breaking bug in heartbeatLockMaterializationRebuild
> ---
>
> Key: HIVE-28190
> URL: https://issues.apache.org/jira/browse/HIVE-28190
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Zsolt Miskolczi
>Assignee: Zsolt Miskolczi
>Priority: Critical
> Labels: pull-request-available
>
> It fails with the following error:
> {code:java}
> org.springframework.dao.InvalidDataAccessApiUsageException: SQL [UPDATE
> "MATERIALIZATION_REBUILD_LOCKS" SET "MRL_LAST_HEARTBEAT" = 1712571919559
> WHERE "MRL_TXN_ID" = 2297 AND "MRL_DB_NAME" = ? AND "MRL_TBL_NAME" = ?]:
> given 2 parameters but expected 0 {code}
> We didn't spot it so far as when the heartbeat of materialized view fails
> with an error, it doesn't affect the rebuild query run. So that it can be
> only spotted by actively watching the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work started] (HIVE-28190) Fix breaking bug in heartbeatLockMaterializationRebuild
[
https://issues.apache.org/jira/browse/HIVE-28190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-28190 started by Zsolt Miskolczi.
--
> Fix breaking bug in heartbeatLockMaterializationRebuild
> ---
>
> Key: HIVE-28190
> URL: https://issues.apache.org/jira/browse/HIVE-28190
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Zsolt Miskolczi
>Assignee: Zsolt Miskolczi
>Priority: Critical
>
> It fails with the following error:
> {code:java}
> org.springframework.dao.InvalidDataAccessApiUsageException: SQL [UPDATE
> "MATERIALIZATION_REBUILD_LOCKS" SET "MRL_LAST_HEARTBEAT" = 1712571919559
> WHERE "MRL_TXN_ID" = 2297 AND "MRL_DB_NAME" = ? AND "MRL_TBL_NAME" = ?]:
> given 2 parameters but expected 0 {code}
> We didn't spot it so far as when the heartbeat of materialized view fails
> with an error, it doesn't affect the rebuild query run. So that it can be
> only spotted by actively watching the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28190) Fix breaking bug in heartbeatLockMaterializationRebuild
Zsolt Miskolczi created HIVE-28190:
--
Summary: Fix breaking bug in heartbeatLockMaterializationRebuild
Key: HIVE-28190
URL: https://issues.apache.org/jira/browse/HIVE-28190
Project: Hive
Issue Type: Bug
Components: Standalone Metastore
Reporter: Zsolt Miskolczi
Assignee: Zsolt Miskolczi
It fails with the following error:
{code:java}
org.springframework.dao.InvalidDataAccessApiUsageException: SQL [UPDATE
"MATERIALIZATION_REBUILD_LOCKS" SET "MRL_LAST_HEARTBEAT" = 1712571919559 WHERE
"MRL_TXN_ID" = 2297 AND "MRL_DB_NAME" = ? AND "MRL_TBL_NAME" = ?]: given 2
parameters but expected 0 {code}
We didn't spot it so far as when the heartbeat of materialized view fails with
an error, it doesn't affect the rebuild query run. So that it can be only
spotted by actively watching the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27965) Table/partition rename takes a long time at PART_COL_STATS for wide tables
[
https://issues.apache.org/jira/browse/HIVE-27965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-27965:
--
Affects Version/s: 4.0.0
> Table/partition rename takes a long time at PART_COL_STATS for wide tables
> --
>
> Key: HIVE-27965
> URL: https://issues.apache.org/jira/browse/HIVE-27965
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Naresh P R
>Priority: Major
>
> Partition table rename gets clogged at PART_COL_STATS for wide tables.
> {code:java}
> CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
> ...
> `DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
> `TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
> `PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
> NULL,
> ...){code}
> Since PART_COL_STATS holds db_name & table_name, incase of table rename,
> every row in PART_COL_STATS associated with the table should be fetched,
> stored in memory, delete & re-insert with new db/table/partition name.
>
> Remove DB_NAME, TABLE_NAME, PARTITION_NAME from PART_COL_STATS, instead use
> TBL_ID, DB_ID, PART_ID to avoid touching PART_COL_STATS for table/partition
> renames.
> Also TBL_ID, DB_ID, PART_ID can be used for PART_COL_STATS INDEXING.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27725) Remove redundant columns in TAB_COL_STATS and PART_COL_STATS
[
https://issues.apache.org/jira/browse/HIVE-27725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17835455#comment-17835455
]
Denys Kuzmenko commented on HIVE-27725:
---
Merged to master.
[~wechar], thanks for the patch, and [~zhangbutao], [~dengzh] for the review!
> Remove redundant columns in TAB_COL_STATS and PART_COL_STATS
>
>
> Key: HIVE-27725
> URL: https://issues.apache.org/jira/browse/HIVE-27725
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0-beta-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
>
> {{TAB_COL_STATS}} table includes {{CAT_NAME}}, {{DB_NAME}} and
> {{TABLE_NAME}}, which can be fetched by join {{TBLS}} and {{DBS}} tables on
> {{TBL_ID}} and {{DB_ID}} columns.
> {{PART_COL_STATS}} table includes {{CAT_NAME}}, {{DB_NAME}}, {{TABLE_NAME}}
> and {{PARTITION_NAME}}, which can be fetched by join {{PARTITIONS}}, {{TBLS}}
> and {{DBS}} tables on {{PART_ID}}, {{TBL_ID}} and {{DB_ID}}.
> In addition, current HMS get table statistics without join other table, while
> delete table statistics with join {{TBLS}}. This inconsistency will result
> exception if in a corner case where some table column statistics were left
> when drop table, then the user recreate the table with same name and database
> name but will get another {{TBL_ID}}, in this case user will get the old
> table column statistics incorrectly. And if user try delete stats fetched by
> get api, the {{NoSuchObjectException}} will be thrown.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-6190) redundant columns in metastore schema for stats
[
https://issues.apache.org/jira/browse/HIVE-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko resolved HIVE-6190.
--
Fix Version/s: 4.1.0
Resolution: Duplicate
> redundant columns in metastore schema for stats
> ---
>
> Key: HIVE-6190
> URL: https://issues.apache.org/jira/browse/HIVE-6190
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Fix For: 4.1.0
>
>
> package.jdo has:
> {noformat}
>
> allows-null="false"/>
>
>
> allows-null="false"/>
>
>
> allows-null="false"/>
>
>
>
>
> {noformat}
> Only PART_ID is enough, the other fields are unnecessary and may potentially
> cause bugs; similarly for table stats. One could argue that they were
> intended for perf (denormalization), but stats retrieval currently is very
> slow on much deeper level so it's not really justified.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27725) Remove redundant columns in TAB_COL_STATS and PART_COL_STATS
[
https://issues.apache.org/jira/browse/HIVE-27725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko resolved HIVE-27725.
---
Fix Version/s: 4.1.0
Resolution: Fixed
> Remove redundant columns in TAB_COL_STATS and PART_COL_STATS
>
>
> Key: HIVE-27725
> URL: https://issues.apache.org/jira/browse/HIVE-27725
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0-beta-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> {{TAB_COL_STATS}} table includes {{CAT_NAME}}, {{DB_NAME}} and
> {{TABLE_NAME}}, which can be fetched by join {{TBLS}} and {{DBS}} tables on
> {{TBL_ID}} and {{DB_ID}} columns.
> {{PART_COL_STATS}} table includes {{CAT_NAME}}, {{DB_NAME}}, {{TABLE_NAME}}
> and {{PARTITION_NAME}}, which can be fetched by join {{PARTITIONS}}, {{TBLS}}
> and {{DBS}} tables on {{PART_ID}}, {{TBL_ID}} and {{DB_ID}}.
> In addition, current HMS get table statistics without join other table, while
> delete table statistics with join {{TBLS}}. This inconsistency will result
> exception if in a corner case where some table column statistics were left
> when drop table, then the user recreate the table with same name and database
> name but will get another {{TBL_ID}}, in this case user will get the old
> table column statistics incorrectly. And if user try delete stats fetched by
> get api, the {{NoSuchObjectException}} will be thrown.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27725) Remove redundant columns in TAB_COL_STATS and PART_COL_STATS
[
https://issues.apache.org/jira/browse/HIVE-27725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-27725:
--
Affects Version/s: 4.0.0
> Remove redundant columns in TAB_COL_STATS and PART_COL_STATS
>
>
> Key: HIVE-27725
> URL: https://issues.apache.org/jira/browse/HIVE-27725
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0, 4.0.0-beta-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> {{TAB_COL_STATS}} table includes {{CAT_NAME}}, {{DB_NAME}} and
> {{TABLE_NAME}}, which can be fetched by join {{TBLS}} and {{DBS}} tables on
> {{TBL_ID}} and {{DB_ID}} columns.
> {{PART_COL_STATS}} table includes {{CAT_NAME}}, {{DB_NAME}}, {{TABLE_NAME}}
> and {{PARTITION_NAME}}, which can be fetched by join {{PARTITIONS}}, {{TBLS}}
> and {{DBS}} tables on {{PART_ID}}, {{TBL_ID}} and {{DB_ID}}.
> In addition, current HMS get table statistics without join other table, while
> delete table statistics with join {{TBLS}}. This inconsistency will result
> exception if in a corner case where some table column statistics were left
> when drop table, then the user recreate the table with same name and database
> name but will get another {{TBL_ID}}, in this case user will get the old
> table column statistics incorrectly. And if user try delete stats fetched by
> get api, the {{NoSuchObjectException}} will be thrown.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work started] (HIVE-27742) LOCAL timezone value is treated as UTC instead of system's timezone which causes data consistency issues
[
https://issues.apache.org/jira/browse/HIVE-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-27742 started by Zoltán Rátkai.
> LOCAL timezone value is treated as UTC instead of system's timezone which
> causes data consistency issues
>
>
> Key: HIVE-27742
> URL: https://issues.apache.org/jira/browse/HIVE-27742
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 4.0.0-beta-1
>Reporter: Janos Kovacs
>Assignee: Zoltán Rátkai
>Priority: Major
>
> The Hive configuration states:
> {noformat}
> HIVE_LOCAL_TIME_ZONE("hive.local.time.zone", "LOCAL",
> "Sets the time-zone for displaying and interpreting time stamps. If
> this property value is set to\n" +
> "LOCAL, it is not specified, or it is not a correct time-zone, the
> system default time-zone will be\n " +
> "used instead. Time-zone IDs can be specified as region-based zone
> IDs (based on IANA time-zone data),\n" +
> "abbreviated zone IDs, or offset IDs."),
> {noformat}
> But seems like in hive4 (-beta) it always treated as UTC - as any other
> invalid timezone value (see HIVE-27741).
> Repro code:
> {noformat}
> docker rm -f hive4
> export HIVE_VERSION=4.0.0-beta-2-SNAPSHOT
> export HS2_ENV_TZ="Europe/Budapest"
> export HS2_USER_TZ=${HS2_ENV_TZ}
> export HIVE_LOCAL_TZ=${HS2_ENV_TZ}
> export HS2_OPTS="-Duser.timezone=$HS2_USER_TZ
> -Dhive.local.time.zone=$HIVE_LOCAL_TZ"
> export HS2_OPTS="$HS2_OPTS
> -Dhive.server2.tez.initialize.default.sessions=false"
> docker run -d -p 1:1 -p 10001:10001 -p 10002:10002 --env
> TZ=${HS2_ENV_TZ} --env SERVICE_OPTS=${HS2_OPTS} --env
> SERVICE_NAME=hiveserver2 --name hive4 apache/hive:${HIVE_VERSION}
> docker exec -it hive4 beeline -u 'jdbc:hive2://localhost:1/' -e "
> SELECT '\${env:TZ}' as \`env:TZ\`,
>'\${system:user.timezone}' as \`system:user.timezone\`,
>'\${hiveconf:hive.local.time.zone}' as
> \`hiveconf:hive.local.time.zone\`;
> DROP TABLE IF EXISTS timestamptest;
> CREATE TABLE timestamptest (
> ts timestamp,
> tz timestamp with local time zone
> ) STORED AS TEXTFILE;
> INSERT INTO timestamptest select TIMESTAMP'2016-01-03
> 12:26:34',TIMESTAMPLOCALTZ'2016-01-03 12:26:34 America/Los_Angeles';
> SET hive.query.results.cache.enabled=false;
> SET hive.local.time.zone=LOCAL;
> SELECT '\${env:TZ}' as \`env:TZ\`,
>'\${system:user.timezone}' as \`system:user.timezone\`,
>'\${hiveconf:hive.local.time.zone}' as
> \`hiveconf:hive.local.time.zone\`;
> SELECT
> 'LOCAL' as
> tzset,
> tz as orig,
> to_utc_timestamp(tz, 'LOCAL') as
> utc_local,
> to_utc_timestamp(tz, 'Europe/Budapest') as
> utc_tz,
> from_utc_timestamp(to_utc_timestamp(tz,'LOCAL'),'Europe/Budapest') as to_bp
> FROM timestamptest;
> SET hive.local.time.zone=Europe/Budapest;
> SELECT '\${env:TZ}' as \`env:TZ\`,
>'\${system:user.timezone}' as \`system:user.timezone\`,
>'\${hiveconf:hive.local.time.zone}' as
> \`hiveconf:hive.local.time.zone\`;
> SELECT
> 'Europe/Budapest'
> as tzset,
> tz
> as orig,
> to_utc_timestamp(tz, 'LOCAL')
> as utc_local,
> to_utc_timestamp(tz, 'Europe/Budapest')
> as utc_tz,
>
> from_utc_timestamp(to_utc_timestamp(tz,'Europe/Budapest'),'Europe/Budapest')
> as to_bp
> FROM timestamptest;
> "
> {noformat}
> The results are:
> {noformat}
> +--+---++
> | env:tz | system:user.timezone | hiveconf:hive.local.time.zone |
> +--+---++
> | Europe/Budapest | Europe/Budapest | LOCAL |
> +--+---++
> ++++++
> | tzset | orig | utc_local|
>utc_tz | to_bp |
> ++++++
> | LOCAL | 2016-01-03 21:26:34.0 Europe/Budapest | 2016-01-03 21:26:34.0 |
> 2016-01-03 20:26:34.0 | 2016-01-03 22:26:34.0 |
> +++-
[jira] [Resolved] (HIVE-28126) Use added record count in cost model when rebuilding materialized view stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-28126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa resolved HIVE-28126.
---
Fix Version/s: 4.1.0
Resolution: Fixed
Merged to master. Thanks [~dkuzmenko] and [~okumin] for review.
> Use added record count in cost model when rebuilding materialized view
> stored by iceberg
> -
>
> Key: HIVE-28126
> URL: https://issues.apache.org/jira/browse/HIVE-28126
> Project: Hive
> Issue Type: Sub-task
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> There is a cost based decision when rebuilding materialized views having
> aggregate.
> Currently the cost model has the total count of rows of source table however
> incremental rebuild plan scans only the rows inserted since the last MV
> rebuild.
> The goal is to update the row count in the cost model in case of Iceberg
> source tables with the sum of the {{added-records}} stored in snapshots
> summary since the last Mv rebuild.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28126) Use added record count in cost model when rebuilding materialized view stored by iceberg
[
https://issues.apache.org/jira/browse/HIVE-28126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-28126:
--
Labels: pull-request-available (was: )
> Use added record count in cost model when rebuilding materialized view
> stored by iceberg
> -
>
> Key: HIVE-28126
> URL: https://issues.apache.org/jira/browse/HIVE-28126
> Project: Hive
> Issue Type: Sub-task
> Components: Iceberg integration, Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
>
> There is a cost based decision when rebuilding materialized views having
> aggregate.
> Currently the cost model has the total count of rows of source table however
> incremental rebuild plan scans only the rows inserted since the last MV
> rebuild.
> The goal is to update the row count in the cost model in case of Iceberg
> source tables with the sum of the {{added-records}} stored in snapshots
> summary since the last Mv rebuild.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27741) Invalid timezone value in to_utc_timestamp() is treated as UTC which can lead to data consistency issues
[
https://issues.apache.org/jira/browse/HIVE-27741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-27741:
--
Affects Version/s: 4.0.0
> Invalid timezone value in to_utc_timestamp() is treated as UTC which can lead
> to data consistency issues
>
>
> Key: HIVE-27741
> URL: https://issues.apache.org/jira/browse/HIVE-27741
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 4.0.0, 4.0.0-beta-1
>Reporter: Janos Kovacs
>Assignee: Zoltán Rátkai
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> When the timezone specified in the *to_utc_timestamp()* function is not
> valid, it still treated as UTC instead of throwing an error. If the user
> accidentally made a typo - e.g. America/Los{color:#ff}*t*{color}_Angeles,
> the query runs successfully returning an invalid converted value which can
> lead to data consistency issues.
> Repro code:
> {noformat}
> docker rm -f hive4
> export HIVE_VERSION=4.0.0-beta-2-SNAPSHOT
> export HS2_ENV_TZ="Europe/Budapest"
> export HS2_USER_TZ=${HS2_ENV_TZ}
> export HIVE_LOCAL_TZ="America/Los_Angeles"
> export HS2_OPTS="-Duser.timezone=$HS2_USER_TZ
> -Dhive.local.time.zone=$HIVE_LOCAL_TZ"
> export HS2_OPTS="$HS2_OPTS
> -Dhive.server2.tez.initialize.default.sessions=false"
> docker run -d -p 1:1 -p 10001:10001 -p 10002:10002 --env
> TZ=${HS2_ENV_TZ} --env SERVICE_OPTS=${HS2_OPTS} --env
> SERVICE_NAME=hiveserver2 --name hive4 apache/hive:${HIVE_VERSION}
> docker exec -it hive4 beeline -u 'jdbc:hive2://localhost:1/' -e "
> SELECT '\${env:TZ}' as \`env:TZ\`,
>'\${system:user.timezone}' as \`system:user.timezone\`,
>'\${hiveconf:hive.local.time.zone}' as
> \`hiveconf:hive.local.time.zone\`;
> DROP TABLE IF EXISTS timestamptest;
> CREATE TABLE timestamptest (
> ts timestamp,
> tz timestamp with local time zone
> ) STORED AS TEXTFILE;
> INSERT INTO timestamptest select TIMESTAMP'2016-01-03
> 12:26:34',TIMESTAMPLOCALTZ'2016-01-03 12:26:34 America/Los_Angeles';
> SELECT
> tzas orig,
> to_utc_timestamp(tz, 'America/Los_Angeles') as utc_correct_tz,
> to_utc_timestamp(tz, 'Europe/HereIsATypo')as utc_incorrect_tz,
> to_utc_timestamp(tz, 'LOCAL') as
> utc_local_aslo_incorrect_tz,
> to_utc_timestamp(tz, 'UTC') as utc_tz
> FROM timestamptest;
> "
> {noformat}
>
> The results are:
> {noformat}
> +--+---++
> | env:tz | system:user.timezone | hiveconf:hive.local.time.zone |
> +--+---++
> | Europe/Budapest | Europe/Budapest | America/Los_Angeles|
> +--+---++
> ++++--++
> |orig| utc_correct_tz |
> utc_incorrect_tz| utc_local_aslo_incorrect_tz | utc_tz |
> ++++--++
> | 2016-01-03 12:26:34.0 America/Los_Angeles | 2016-01-03 20:26:34.0 |
> 2016-01-03 12:26:34.0 | 2016-01-03 12:26:34.0| 2016-01-03 12:26:34.0
> |
> ++++--++
> {noformat}
> Note:
> * the invalid timezone - utc_incorrect_tz - is treated as UTC
> * also note that LOCAL is also treated as UTC which in fact should be
> treated as system's timezone, but as LOCAL is also an invalid timezone value
> in hive4, ut becomes UTC just like any other invalid and/or typo timezone
> values (see HIVE-27742)
>
> Hive should throw an Exception in that case to let the user know that the
> provided timezone is wrong - at least this should be configurable, e.g. via
> something like {*}hive.strict.time.zone.check{*}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27741) Invalid timezone value in to_utc_timestamp() is treated as UTC which can lead to data consistency issues
[
https://issues.apache.org/jira/browse/HIVE-27741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17835351#comment-17835351
]
Denys Kuzmenko commented on HIVE-27741:
---
Merged to master
[~zratkai] thanks for the patch and [~okumin], [~simhadri-g] for the review!
> Invalid timezone value in to_utc_timestamp() is treated as UTC which can lead
> to data consistency issues
>
>
> Key: HIVE-27741
> URL: https://issues.apache.org/jira/browse/HIVE-27741
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 4.0.0-beta-1
>Reporter: Janos Kovacs
>Assignee: Zoltán Rátkai
>Priority: Major
> Labels: pull-request-available
>
> When the timezone specified in the *to_utc_timestamp()* function is not
> valid, it still treated as UTC instead of throwing an error. If the user
> accidentally made a typo - e.g. America/Los{color:#ff}*t*{color}_Angeles,
> the query runs successfully returning an invalid converted value which can
> lead to data consistency issues.
> Repro code:
> {noformat}
> docker rm -f hive4
> export HIVE_VERSION=4.0.0-beta-2-SNAPSHOT
> export HS2_ENV_TZ="Europe/Budapest"
> export HS2_USER_TZ=${HS2_ENV_TZ}
> export HIVE_LOCAL_TZ="America/Los_Angeles"
> export HS2_OPTS="-Duser.timezone=$HS2_USER_TZ
> -Dhive.local.time.zone=$HIVE_LOCAL_TZ"
> export HS2_OPTS="$HS2_OPTS
> -Dhive.server2.tez.initialize.default.sessions=false"
> docker run -d -p 1:1 -p 10001:10001 -p 10002:10002 --env
> TZ=${HS2_ENV_TZ} --env SERVICE_OPTS=${HS2_OPTS} --env
> SERVICE_NAME=hiveserver2 --name hive4 apache/hive:${HIVE_VERSION}
> docker exec -it hive4 beeline -u 'jdbc:hive2://localhost:1/' -e "
> SELECT '\${env:TZ}' as \`env:TZ\`,
>'\${system:user.timezone}' as \`system:user.timezone\`,
>'\${hiveconf:hive.local.time.zone}' as
> \`hiveconf:hive.local.time.zone\`;
> DROP TABLE IF EXISTS timestamptest;
> CREATE TABLE timestamptest (
> ts timestamp,
> tz timestamp with local time zone
> ) STORED AS TEXTFILE;
> INSERT INTO timestamptest select TIMESTAMP'2016-01-03
> 12:26:34',TIMESTAMPLOCALTZ'2016-01-03 12:26:34 America/Los_Angeles';
> SELECT
> tzas orig,
> to_utc_timestamp(tz, 'America/Los_Angeles') as utc_correct_tz,
> to_utc_timestamp(tz, 'Europe/HereIsATypo')as utc_incorrect_tz,
> to_utc_timestamp(tz, 'LOCAL') as
> utc_local_aslo_incorrect_tz,
> to_utc_timestamp(tz, 'UTC') as utc_tz
> FROM timestamptest;
> "
> {noformat}
>
> The results are:
> {noformat}
> +--+---++
> | env:tz | system:user.timezone | hiveconf:hive.local.time.zone |
> +--+---++
> | Europe/Budapest | Europe/Budapest | America/Los_Angeles|
> +--+---++
> ++++--++
> |orig| utc_correct_tz |
> utc_incorrect_tz| utc_local_aslo_incorrect_tz | utc_tz |
> ++++--++
> | 2016-01-03 12:26:34.0 America/Los_Angeles | 2016-01-03 20:26:34.0 |
> 2016-01-03 12:26:34.0 | 2016-01-03 12:26:34.0| 2016-01-03 12:26:34.0
> |
> ++++--++
> {noformat}
> Note:
> * the invalid timezone - utc_incorrect_tz - is treated as UTC
> * also note that LOCAL is also treated as UTC which in fact should be
> treated as system's timezone, but as LOCAL is also an invalid timezone value
> in hive4, ut becomes UTC just like any other invalid and/or typo timezone
> values (see HIVE-27742)
>
> Hive should throw an Exception in that case to let the user know that the
> provided timezone is wrong - at least this should be configurable, e.g. via
> something like {*}hive.strict.time.zone.check{*}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27741) Invalid timezone value in to_utc_timestamp() is treated as UTC which can lead to data consistency issues
[
https://issues.apache.org/jira/browse/HIVE-27741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko resolved HIVE-27741.
---
Fix Version/s: 4.1.0
Target Version/s: 4.1.0 (was: 4.0.0)
Resolution: Fixed
> Invalid timezone value in to_utc_timestamp() is treated as UTC which can lead
> to data consistency issues
>
>
> Key: HIVE-27741
> URL: https://issues.apache.org/jira/browse/HIVE-27741
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 4.0.0-beta-1
>Reporter: Janos Kovacs
>Assignee: Zoltán Rátkai
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> When the timezone specified in the *to_utc_timestamp()* function is not
> valid, it still treated as UTC instead of throwing an error. If the user
> accidentally made a typo - e.g. America/Los{color:#ff}*t*{color}_Angeles,
> the query runs successfully returning an invalid converted value which can
> lead to data consistency issues.
> Repro code:
> {noformat}
> docker rm -f hive4
> export HIVE_VERSION=4.0.0-beta-2-SNAPSHOT
> export HS2_ENV_TZ="Europe/Budapest"
> export HS2_USER_TZ=${HS2_ENV_TZ}
> export HIVE_LOCAL_TZ="America/Los_Angeles"
> export HS2_OPTS="-Duser.timezone=$HS2_USER_TZ
> -Dhive.local.time.zone=$HIVE_LOCAL_TZ"
> export HS2_OPTS="$HS2_OPTS
> -Dhive.server2.tez.initialize.default.sessions=false"
> docker run -d -p 1:1 -p 10001:10001 -p 10002:10002 --env
> TZ=${HS2_ENV_TZ} --env SERVICE_OPTS=${HS2_OPTS} --env
> SERVICE_NAME=hiveserver2 --name hive4 apache/hive:${HIVE_VERSION}
> docker exec -it hive4 beeline -u 'jdbc:hive2://localhost:1/' -e "
> SELECT '\${env:TZ}' as \`env:TZ\`,
>'\${system:user.timezone}' as \`system:user.timezone\`,
>'\${hiveconf:hive.local.time.zone}' as
> \`hiveconf:hive.local.time.zone\`;
> DROP TABLE IF EXISTS timestamptest;
> CREATE TABLE timestamptest (
> ts timestamp,
> tz timestamp with local time zone
> ) STORED AS TEXTFILE;
> INSERT INTO timestamptest select TIMESTAMP'2016-01-03
> 12:26:34',TIMESTAMPLOCALTZ'2016-01-03 12:26:34 America/Los_Angeles';
> SELECT
> tzas orig,
> to_utc_timestamp(tz, 'America/Los_Angeles') as utc_correct_tz,
> to_utc_timestamp(tz, 'Europe/HereIsATypo')as utc_incorrect_tz,
> to_utc_timestamp(tz, 'LOCAL') as
> utc_local_aslo_incorrect_tz,
> to_utc_timestamp(tz, 'UTC') as utc_tz
> FROM timestamptest;
> "
> {noformat}
>
> The results are:
> {noformat}
> +--+---++
> | env:tz | system:user.timezone | hiveconf:hive.local.time.zone |
> +--+---++
> | Europe/Budapest | Europe/Budapest | America/Los_Angeles|
> +--+---++
> ++++--++
> |orig| utc_correct_tz |
> utc_incorrect_tz| utc_local_aslo_incorrect_tz | utc_tz |
> ++++--++
> | 2016-01-03 12:26:34.0 America/Los_Angeles | 2016-01-03 20:26:34.0 |
> 2016-01-03 12:26:34.0 | 2016-01-03 12:26:34.0| 2016-01-03 12:26:34.0
> |
> ++++--++
> {noformat}
> Note:
> * the invalid timezone - utc_incorrect_tz - is treated as UTC
> * also note that LOCAL is also treated as UTC which in fact should be
> treated as system's timezone, but as LOCAL is also an invalid timezone value
> in hive4, ut becomes UTC just like any other invalid and/or typo timezone
> values (see HIVE-27742)
>
> Hive should throw an Exception in that case to let the user know that the
> provided timezone is wrong - at least this should be configurable, e.g. via
> something like {*}hive.strict.time.zone.check{*}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28143) After HIVE-27492 fix, some HPLSQL built-in functions like trim, lower are not working when used in insert statement.
[
https://issues.apache.org/jira/browse/HIVE-28143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-28143:
--
Target Version/s: 4.1.0
> After HIVE-27492 fix, some HPLSQL built-in functions like trim, lower are not
> working when used in insert statement.
>
>
> Key: HIVE-28143
> URL: https://issues.apache.org/jira/browse/HIVE-28143
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Affects Versions: 4.0.0
>Reporter: Dayakar M
>Assignee: Dayakar M
>Priority: Major
> Labels: hive-4.1.0-must, pull-request-available
>
> After HIVE-27492 fix, some HPLSQL built-in functions like trim, lower are not
> working when used in insert statement.
> Steps to reproduce:
> {noformat}
> CREATE TABLE result (name String);
> CREATE PROCEDURE p1(s1 string)
> BEGIN\n" +
> INSERT INTO result VALUES(lower(s1));
> END;
> call p1('abcd');
> SELECT * FROM result;{noformat}
> Error reported:
> {noformat}
> ERROR : Ln:3 identifier 'LOWER' must be declared.{noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28143) After HIVE-27492 fix, some HPLSQL built-in functions like trim, lower are not working when used in insert statement.
[
https://issues.apache.org/jira/browse/HIVE-28143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-28143:
--
Labels: hive-4.1.0-must pull-request-available (was:
pull-request-available)
> After HIVE-27492 fix, some HPLSQL built-in functions like trim, lower are not
> working when used in insert statement.
>
>
> Key: HIVE-28143
> URL: https://issues.apache.org/jira/browse/HIVE-28143
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Affects Versions: 4.0.0
>Reporter: Dayakar M
>Assignee: Dayakar M
>Priority: Major
> Labels: hive-4.1.0-must, pull-request-available
>
> After HIVE-27492 fix, some HPLSQL built-in functions like trim, lower are not
> working when used in insert statement.
> Steps to reproduce:
> {noformat}
> CREATE TABLE result (name String);
> CREATE PROCEDURE p1(s1 string)
> BEGIN\n" +
> INSERT INTO result VALUES(lower(s1));
> END;
> call p1('abcd');
> SELECT * FROM result;{noformat}
> Error reported:
> {noformat}
> ERROR : Ln:3 identifier 'LOWER' must be declared.{noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28189) PrivilegeSynchonizer Need to lock the table while synchronizing privs data
[ https://issues.apache.org/jira/browse/HIVE-28189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuping Fan updated HIVE-28189: -- Description: Due to the large number of permissions, the synchronization time exceeds 1 minute. If I delete this table during that 1 minute, it will cause the table to fail to delete, with the error that the tbl_col_privs foreign key constraint check failed. So I think the table should be locked when the permissions are synchronized > PrivilegeSynchonizer Need to lock the table while synchronizing privs data > -- > > Key: HIVE-28189 > URL: https://issues.apache.org/jira/browse/HIVE-28189 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Yuping Fan >Priority: Major > > Due to the large number of permissions, the synchronization time exceeds 1 > minute. > If I delete this table during that 1 minute, it will cause the table to fail > to delete, with the error that the tbl_col_privs foreign key constraint check > failed. So I think the table should be locked when the permissions are > synchronized -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28189) PrivilegeSynchonizer Need to lock the table while synchronizing privs data
Yuping Fan created HIVE-28189: - Summary: PrivilegeSynchonizer Need to lock the table while synchronizing privs data Key: HIVE-28189 URL: https://issues.apache.org/jira/browse/HIVE-28189 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 3.1.0 Reporter: Yuping Fan -- This message was sent by Atlassian Jira (v8.20.10#820010)
