[jira] [Work logged] (HIVE-27172) Add the HMS client connection timeout config
[
https://issues.apache.org/jira/browse/HIVE-27172?focusedWorklogId=852946=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852946
]
ASF GitHub Bot logged work on HIVE-27172:
-
Author: ASF GitHub Bot
Created on: 25/Mar/23 04:35
Start Date: 25/Mar/23 04:35
Worklog Time Spent: 10m
Work Description: wecharyu commented on code in PR #4150:
URL: https://github.com/apache/hive/pull/4150#discussion_r1148302834
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##
@@ -402,6 +402,8 @@ public enum ConfVars {
"has an infinite lifetime."),
CLIENT_SOCKET_TIMEOUT("metastore.client.socket.timeout",
"hive.metastore.client.socket.timeout", 600,
TimeUnit.SECONDS, "MetaStore Client socket timeout in seconds"),
+CLIENT_CONNECTION_TIMEOUT("metastore.client.connection.timeout",
"hive.metastore.client.connection.timeout", 10,
Review Comment:
Nice catch, if there is any concern I will keep the default connection
timeout value same as socket timeout.
Issue Time Tracking
---
Worklog Id: (was: 852946)
Time Spent: 1h (was: 50m)
> Add the HMS client connection timeout config
>
>
> Key: HIVE-27172
> URL: https://issues.apache.org/jira/browse/HIVE-27172
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently {{HiveMetastoreClient}} use {{CLIENT_SOCKET_TIMEOUT}} as both
> socket timeout and connection timeout, it's not convenient for users to set a
> smaller connection timeout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27150) Drop single partition can also support direct sql
[ https://issues.apache.org/jira/browse/HIVE-27150?focusedWorklogId=852945=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852945 ] ASF GitHub Bot logged work on HIVE-27150: - Author: ASF GitHub Bot Created on: 25/Mar/23 04:28 Start Date: 25/Mar/23 04:28 Worklog Time Spent: 10m Work Description: wecharyu commented on code in PR #4123: URL: https://github.com/apache/hive/pull/4123#discussion_r1148297352 ## standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java: ## @@ -459,16 +459,15 @@ boolean doesPartitionExist(String catName, String dbName, String tableName, * @param catName catalog name. * @param dbName database name. * @param tableName table name. - * @param part_vals list of partition values. + * @param partName partition name. * @return true if the partition was dropped. * @throws MetaException Error accessing the RDBMS. * @throws NoSuchObjectException no partition matching this description exists * @throws InvalidObjectException error dropping the statistics for the partition * @throws InvalidInputException error dropping the statistics for the partition */ - boolean dropPartition(String catName, String dbName, String tableName, - List part_vals) throws MetaException, NoSuchObjectException, InvalidObjectException, - InvalidInputException; + boolean dropPartition(String catName, String dbName, String tableName, String partName) Review Comment: Make sense, my initial thought was that this is an internal API which is not exposed to clients, so compatibility was not considered. I will follow your comment. Issue Time Tracking --- Worklog Id: (was: 852945) Time Spent: 1h (was: 50m) > Drop single partition can also support direct sql > - > > Key: HIVE-27150 > URL: https://issues.apache.org/jira/browse/HIVE-27150 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > *Background:* > [HIVE-6980|https://issues.apache.org/jira/browse/HIVE-6980] supports direct > sql for drop_partitions, we can reuse this huge improvement in drop_partition. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27154) Fix testBootstrapReplLoadRetryAfterFailureForPartitions
[ https://issues.apache.org/jira/browse/HIVE-27154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar resolved HIVE-27154. Fix Version/s: 3.2.0 Resolution: Fixed > Fix testBootstrapReplLoadRetryAfterFailureForPartitions > --- > > Key: HIVE-27154 > URL: https://issues.apache.org/jira/browse/HIVE-27154 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > `testBootstrapReplLoadRetryAfterFailureForPartitions` has been failing on > branch-3 > > http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-27154) Fix testBootstrapReplLoadRetryAfterFailureForPartitions
[ https://issues.apache.org/jira/browse/HIVE-27154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-27154 started by Vihang Karajgaonkar. -- > Fix testBootstrapReplLoadRetryAfterFailureForPartitions > --- > > Key: HIVE-27154 > URL: https://issues.apache.org/jira/browse/HIVE-27154 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > `testBootstrapReplLoadRetryAfterFailureForPartitions` has been failing on > branch-3 > > http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27154) Fix testBootstrapReplLoadRetryAfterFailureForPartitions
[ https://issues.apache.org/jira/browse/HIVE-27154?focusedWorklogId=852940=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852940 ] ASF GitHub Bot logged work on HIVE-27154: - Author: ASF GitHub Bot Created on: 25/Mar/23 03:19 Start Date: 25/Mar/23 03:19 Worklog Time Spent: 10m Work Description: vihangk1 merged PR #4151: URL: https://github.com/apache/hive/pull/4151 Issue Time Tracking --- Worklog Id: (was: 852940) Time Spent: 1h 10m (was: 1h) > Fix testBootstrapReplLoadRetryAfterFailureForPartitions > --- > > Key: HIVE-27154 > URL: https://issues.apache.org/jira/browse/HIVE-27154 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > `testBootstrapReplLoadRetryAfterFailureForPartitions` has been failing on > branch-3 > > http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work stopped] (HIVE-27154) Fix testBootstrapReplLoadRetryAfterFailureForPartitions
[ https://issues.apache.org/jira/browse/HIVE-27154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-27154 stopped by Vihang Karajgaonkar. -- > Fix testBootstrapReplLoadRetryAfterFailureForPartitions > --- > > Key: HIVE-27154 > URL: https://issues.apache.org/jira/browse/HIVE-27154 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > `testBootstrapReplLoadRetryAfterFailureForPartitions` has been failing on > branch-3 > > http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26400) Provide docker images for Hive
[ https://issues.apache.org/jira/browse/HIVE-26400?focusedWorklogId=852939=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852939 ] ASF GitHub Bot logged work on HIVE-26400: - Author: ASF GitHub Bot Created on: 25/Mar/23 02:09 Start Date: 25/Mar/23 02:09 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3448: URL: https://github.com/apache/hive/pull/3448#issuecomment-1483681136 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3448) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3448=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3448=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3448=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3448=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3448=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 852939) Time Spent: 10h 10m (was: 10h) > Provide docker images for Hive > -- > > Key: HIVE-26400 > URL: https://issues.apache.org/jira/browse/HIVE-26400 > Project: Hive > Issue Type: Sub-task > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Blocker > Labels: hive-4.0.0-must, pull-request-available > Time Spent: 10h 10m > Remaining Estimate: 0h > > Make Apache Hive be able to run inside docker container in pseudo-distributed > mode, with MySQL/Derby as its back database, provide the following: > * Quick-start/Debugging/Prepare a test env for Hive; > * Tools to build target image with specified version of Hive and its > dependencies; > * Images can be used as the basis for the Kubernetes operator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27158) Store hive columns stats in puffin files for iceberg tables
[ https://issues.apache.org/jira/browse/HIVE-27158?focusedWorklogId=852938=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852938 ] ASF GitHub Bot logged work on HIVE-27158: - Author: ASF GitHub Bot Created on: 25/Mar/23 02:05 Start Date: 25/Mar/23 02:05 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4131: URL: https://github.com/apache/hive/pull/4131#issuecomment-1483680009 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4131) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4131=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4131=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4131=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=CODE_SMELL) [6 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4131=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4131=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 852938) Time Spent: 1h 20m (was: 1h 10m) > Store hive columns stats in puffin files for iceberg tables > --- > > Key: HIVE-27158 > URL: https://issues.apache.org/jira/browse/HIVE-27158 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26400) Provide docker images for Hive
[ https://issues.apache.org/jira/browse/HIVE-26400?focusedWorklogId=852935=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852935 ] ASF GitHub Bot logged work on HIVE-26400: - Author: ASF GitHub Bot Created on: 25/Mar/23 00:44 Start Date: 25/Mar/23 00:44 Worklog Time Spent: 10m Work Description: dengzhhu653 closed pull request #4133: HIVE-26400: Provide docker image for Hive URL: https://github.com/apache/hive/pull/4133 Issue Time Tracking --- Worklog Id: (was: 852935) Time Spent: 10h (was: 9h 50m) > Provide docker images for Hive > -- > > Key: HIVE-26400 > URL: https://issues.apache.org/jira/browse/HIVE-26400 > Project: Hive > Issue Type: Sub-task > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Blocker > Labels: hive-4.0.0-must, pull-request-available > Time Spent: 10h > Remaining Estimate: 0h > > Make Apache Hive be able to run inside docker container in pseudo-distributed > mode, with MySQL/Derby as its back database, provide the following: > * Quick-start/Debugging/Prepare a test env for Hive; > * Tools to build target image with specified version of Hive and its > dependencies; > * Images can be used as the basis for the Kubernetes operator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27150) Drop single partition can also support direct sql
[ https://issues.apache.org/jira/browse/HIVE-27150?focusedWorklogId=852926=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852926 ] ASF GitHub Bot logged work on HIVE-27150: - Author: ASF GitHub Bot Created on: 24/Mar/23 21:55 Start Date: 24/Mar/23 21:55 Worklog Time Spent: 10m Work Description: deniskuzZ commented on code in PR #4123: URL: https://github.com/apache/hive/pull/4123#discussion_r1148088843 ## standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java: ## @@ -459,16 +459,15 @@ boolean doesPartitionExist(String catName, String dbName, String tableName, * @param catName catalog name. * @param dbName database name. * @param tableName table name. - * @param part_vals list of partition values. + * @param partName partition name. * @return true if the partition was dropped. * @throws MetaException Error accessing the RDBMS. * @throws NoSuchObjectException no partition matching this description exists * @throws InvalidObjectException error dropping the statistics for the partition * @throws InvalidInputException error dropping the statistics for the partition */ - boolean dropPartition(String catName, String dbName, String tableName, - List part_vals) throws MetaException, NoSuchObjectException, InvalidObjectException, - InvalidInputException; + boolean dropPartition(String catName, String dbName, String tableName, String partName) Review Comment: That is a change in API that breaks backward compatibility. We should keep the original signature and mark it as deprecated. In the following releases we could remove it. Issue Time Tracking --- Worklog Id: (was: 852926) Time Spent: 50m (was: 40m) > Drop single partition can also support direct sql > - > > Key: HIVE-27150 > URL: https://issues.apache.org/jira/browse/HIVE-27150 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > *Background:* > [HIVE-6980|https://issues.apache.org/jira/browse/HIVE-6980] supports direct > sql for drop_partitions, we can reuse this huge improvement in drop_partition. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27150) Drop single partition can also support direct sql
[ https://issues.apache.org/jira/browse/HIVE-27150?focusedWorklogId=852925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852925 ] ASF GitHub Bot logged work on HIVE-27150: - Author: ASF GitHub Bot Created on: 24/Mar/23 21:51 Start Date: 24/Mar/23 21:51 Worklog Time Spent: 10m Work Description: deniskuzZ commented on code in PR #4123: URL: https://github.com/apache/hive/pull/4123#discussion_r1148088843 ## standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java: ## @@ -459,16 +459,15 @@ boolean doesPartitionExist(String catName, String dbName, String tableName, * @param catName catalog name. * @param dbName database name. * @param tableName table name. - * @param part_vals list of partition values. + * @param partName partition name. * @return true if the partition was dropped. * @throws MetaException Error accessing the RDBMS. * @throws NoSuchObjectException no partition matching this description exists * @throws InvalidObjectException error dropping the statistics for the partition * @throws InvalidInputException error dropping the statistics for the partition */ - boolean dropPartition(String catName, String dbName, String tableName, - List part_vals) throws MetaException, NoSuchObjectException, InvalidObjectException, - InvalidInputException; + boolean dropPartition(String catName, String dbName, String tableName, String partName) Review Comment: That is a change in API that breaks backward compatibility. Issue Time Tracking --- Worklog Id: (was: 852925) Time Spent: 40m (was: 0.5h) > Drop single partition can also support direct sql > - > > Key: HIVE-27150 > URL: https://issues.apache.org/jira/browse/HIVE-27150 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > *Background:* > [HIVE-6980|https://issues.apache.org/jira/browse/HIVE-6980] supports direct > sql for drop_partitions, we can reuse this huge improvement in drop_partition. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27167) Upgrade guava version in standalone-metastore and storage-api module
[ https://issues.apache.org/jira/browse/HIVE-27167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17704824#comment-17704824 ] Denys Kuzmenko commented on HIVE-27167: --- Merged to master. Thank you for the patch [~Aggarwal_Raghav] and [~cnauroth] for the review! > Upgrade guava version in standalone-metastore and storage-api module > > > Key: HIVE-27167 > URL: https://issues.apache.org/jira/browse/HIVE-27167 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, storage-api >Affects Versions: 4.0.0-alpha-2 >Reporter: Raghav Aggarwal >Assignee: Raghav Aggarwal >Priority: Minor > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The guava version in standalone-metastore and storage-api (i.e 19.0) is not > in sync with the the parent pom.xml (i.e 22.0). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27167) Upgrade guava version in standalone-metastore and storage-api module
[ https://issues.apache.org/jira/browse/HIVE-27167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko resolved HIVE-27167. --- Fix Version/s: 4.0.0 Resolution: Fixed > Upgrade guava version in standalone-metastore and storage-api module > > > Key: HIVE-27167 > URL: https://issues.apache.org/jira/browse/HIVE-27167 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, storage-api >Affects Versions: 4.0.0-alpha-2 >Reporter: Raghav Aggarwal >Assignee: Raghav Aggarwal >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The guava version in standalone-metastore and storage-api (i.e 19.0) is not > in sync with the the parent pom.xml (i.e 22.0). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27167) Upgrade guava version in standalone-metastore and storage-api module
[ https://issues.apache.org/jira/browse/HIVE-27167?focusedWorklogId=852924=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852924 ] ASF GitHub Bot logged work on HIVE-27167: - Author: ASF GitHub Bot Created on: 24/Mar/23 21:40 Start Date: 24/Mar/23 21:40 Worklog Time Spent: 10m Work Description: deniskuzZ merged PR #4143: URL: https://github.com/apache/hive/pull/4143 Issue Time Tracking --- Worklog Id: (was: 852924) Time Spent: 40m (was: 0.5h) > Upgrade guava version in standalone-metastore and storage-api module > > > Key: HIVE-27167 > URL: https://issues.apache.org/jira/browse/HIVE-27167 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, storage-api >Affects Versions: 4.0.0-alpha-2 >Reporter: Raghav Aggarwal >Assignee: Raghav Aggarwal >Priority: Minor > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The guava version in standalone-metastore and storage-api (i.e 19.0) is not > in sync with the the parent pom.xml (i.e 22.0). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27172) Add the HMS client connection timeout config
[
https://issues.apache.org/jira/browse/HIVE-27172?focusedWorklogId=852921=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852921
]
ASF GitHub Bot logged work on HIVE-27172:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 20:51
Start Date: 24/Mar/23 20:51
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #4150:
URL: https://github.com/apache/hive/pull/4150#issuecomment-1483391646
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4150)
[](https://sonarcloud.io/project/issues?id=apache_hive=4150=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=4150=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4150=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=4150=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=4150=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4150=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4150=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4150=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4150=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=4150=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=4150=false=CODE_SMELL)
[3 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=4150=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=4150=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=4150=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 852921)
Time Spent: 50m (was: 40m)
> Add the HMS client connection timeout config
>
>
> Key: HIVE-27172
> URL: https://issues.apache.org/jira/browse/HIVE-27172
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Currently {{HiveMetastoreClient}} use {{CLIENT_SOCKET_TIMEOUT}} as both
> socket timeout and connection timeout, it's not convenient for users to set a
> smaller connection timeout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27151) Revert "HIVE-21685 Wrong simplification in query with multiple IN clauses"
[
https://issues.apache.org/jira/browse/HIVE-27151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Raj updated HIVE-27151:
Fix Version/s: 3.2.0
> Revert "HIVE-21685 Wrong simplification in query with multiple IN clauses"
> --
>
> Key: HIVE-27151
> URL: https://issues.apache.org/jira/browse/HIVE-27151
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The multi_in_clause.q fails because Hive is not able to parse
> explain cbo
> select * from very_simple_table_for_in_test where name IN('g','r') AND name
> IN('a','b')
> If we want this to work, I am able to do it in my local. We have 2 options :
> a. Either revert HIVE-21685 since this scenario was not validated back then
> before adding this test.
> b. This fix was present in https://issues.apache.org/jira/browse/HIVE-20718
> but to cherry pick this we need to cherry pick
> https://issues.apache.org/jira/browse/HIVE-17040 since HIVE-20718 has a lot
> of merge conflicts with HIVE-17040. But after cherry picking these we have
> other failures to fix.
>
> I am reverting this ticket for now.
> Exception stacktrace :
> {code:java}
> 2023-03-16 12:33:11 Completed running task attempt:
> attempt_1678994907903_0001_185_01_00_02023-03-16 12:33:11 Completed Dag:
> dag_1678994907903_0001_185TRACE StatusLogger Log4jLoggerFactory.getContext()
> found anchor class org.apache.hadoop.hive.ql.exec.OperatorTRACE StatusLogger
> Log4jLoggerFactory.getContext() found anchor class
> org.apache.hadoop.hive.ql.stats.fs.FSStatsPublisherTRACE StatusLogger
> Log4jLoggerFactory.getContext() found anchor class
> org.apache.hadoop.hive.ql.stats.fs.FSStatsAggregatorNoViableAltException(24@[])
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.explainStatement(HiveParser.java:1512)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1407)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:230)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:79) at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:72) at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:617)at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1854) at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1801) at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1796) at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
>at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1474)
>at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1448)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:177)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> at
> org.apache.hadoop.hive.cli.split12.TestMiniLlapLocalCliDriver.testCliDriver(TestMiniLlapLocalCliDriver.java:62)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27152) Revert "Constant UDF is not pushed to JDBCStorage Handler"
[
https://issues.apache.org/jira/browse/HIVE-27152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Raj updated HIVE-27152:
Fix Version/s: 3.2.0
> Revert "Constant UDF is not pushed to JDBCStorage Handler"
> --
>
> Key: HIVE-27152
> URL: https://issues.apache.org/jira/browse/HIVE-27152
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> current_date_timestamp.q - This change was committed in HIVE-21388 without
> validation.
> The failure is because again Hive is not able to parse
> explain cbo select current_timestamp() from alltypesorc
>
> Exception stack trace :
> {code:java}
> 2023-03-16 04:06:17 Completed running task attempt:
> attempt_1678964507586_0001_175_01_00_02023-03-16 04:06:17 Completed Dag:
> dag_1678964507586_0001_175TRACE StatusLogger Log4jLoggerFactory.getContext()
> found anchor class org.apache.hadoop.hive.ql.exec.OperatorTRACE StatusLogger
> Log4jLoggerFactory.getContext() found anchor class
> org.apache.hadoop.hive.ql.stats.fs.FSStatsPublisherTRACE StatusLogger
> Log4jLoggerFactory.getContext() found anchor class
> org.apache.hadoop.hive.ql.stats.fs.FSStatsAggregatorNoViableAltException(24@[])
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.explainStatement(HiveParser.java:1512)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1407)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:230)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:79) at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:72) at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:617)at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1854) at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1801) at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1796) at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
>at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1474)
>at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1448)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:177)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> at
> org.apache.hadoop.hive.cli.split12.TestMiniLlapLocalCliDriver.testCliDriver(TestMiniLlapLocalCliDriver.java:62)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
>at org.junit.rules.RunRules.evaluate(RunRules.java:20) Attachments
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27148) Disable TestJdbcGenericUDTFGetSplits
[ https://issues.apache.org/jira/browse/HIVE-27148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Raj updated HIVE-27148: Fix Version/s: 3.2.0 > Disable TestJdbcGenericUDTFGetSplits > > > Key: HIVE-27148 > URL: https://issues.apache.org/jira/browse/HIVE-27148 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > TestJdbcGenericUDTFGetSplits is flaky and intermittently fails. > http://ci.hive.apache.org/job/hive-flaky-check/614/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27153) Revert "HIVE-20182: Backport HIVE-20067 to branch-3"
[
https://issues.apache.org/jira/browse/HIVE-27153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Raj updated HIVE-27153:
Fix Version/s: 3.2.0
> Revert "HIVE-20182: Backport HIVE-20067 to branch-3"
>
>
> Key: HIVE-27153
> URL: https://issues.apache.org/jira/browse/HIVE-27153
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The mm_all.q test is failing because of this commit. This commit was not
> validated before committing.
> There is no stack trace for this exception. Link to the exception :
> [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4126/2/tests]
>
> {code:java}
> java.lang.AssertionError: Client execution failed with error code = 1 running
> "insert into table part_mm_n0 partition(key_mm=455) select key from
> intermediate_n0" fname=mm_all.q See ./ql/target/tmp/log/hive.log or
> ./itests/qtest/target/tmp/log/hive.log, or check ./ql/target/surefire-reports
> or ./itests/qtest/target/surefire-reports/ for specific test cases logs. at
> org.junit.Assert.fail(Assert.java:88)at
> org.apache.hadoop.hive.ql.QTestUtil.failed(QTestUtil.java:2232) at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:180)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> at
> org.apache.hadoop.hive.cli.split1.TestMiniLlapCliDriver.testCliDriver(TestMiniLlapCliDriver.java:62)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) {code}
>
>
> Found the actual error :
> {code:java}
> 2023-03-19T15:18:07,705 DEBUG [699603ee-f4a1-43b7-b160-7faf858ca4b4 main]
> converters.ArrayConverter: Converting 'java.net.URL[]' value
> '[Ljava.net.URL;@7535f28' to type 'java.net.URL[]'
> 2023-03-19T15:18:07,705 DEBUG [699603ee-f4a1-43b7-b160-7faf858ca4b4 main]
> converters.ArrayConverter: No conversion required, value is already a
> java.net.URL[]
> 2023-03-19T15:18:07,819 INFO [699603ee-f4a1-43b7-b160-7faf858ca4b4 main]
> beanutils.FluentPropertyBeanIntrospector: Error when creating
> PropertyDescriptor for public final void
> org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
> Ignoring this property.
> 2023-03-19T15:18:07,819 DEBUG [699603ee-f4a1-43b7-b160-7faf858ca4b4 main]
> beanutils.FluentPropertyBeanIntrospector: Exception is:
> java.beans.IntrospectionException: bad write method arg count: public final
> void
> org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)
> at
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:657)
> ~[?:1.8.0_342]
> at
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:327)
> ~[?:1.8.0_342]
> at java.beans.PropertyDescriptor.(PropertyDescriptor.java:139)
> ~[?:1.8.0_342]
> at
> org.apache.commons.beanutils.FluentPropertyBeanIntrospector.createFluentPropertyDescritor(FluentPropertyBeanIntrospector.java:178)
> ~[commons-beanutils-1.9.3.jar:1.9.3]
> at
> org.apache.commons.beanutils.FluentPropertyBeanIntrospector.introspect(FluentPropertyBeanIntrospector.java:141)
> [commons-beanutils-1.9.3.jar:1.9.3]
> at
> org.apache.commons.beanutils.PropertyUtilsBean.fetchIntrospectionData(PropertyUtilsBean.java:2245)
> [commons-beanutils-1.9.3.jar:1.9.3]
> at
> org.apache.commons.beanutils.PropertyUtilsBean.getIntrospectionData(PropertyUtilsBean.java:2226)
> [commons-beanutils-1.9.3.jar:1.9.3]
> at
> org.apache.commons.beanutils.PropertyUtilsBean.getPropertyDescriptor(PropertyUtilsBean.java:954)
> [commons-beanutils-1.9.3.jar:1.9.3]
> at
> org.apache.commons.beanutils.PropertyUtilsBean.isWriteable(PropertyUtilsBean.java:1478)
> [commons-beanutils-1.9.3.jar:1.9.3]
> at
> org.apache.commons.configuration2.beanutils.BeanHelper.isPropertyWriteable(BeanHelper.java:521)
> [commons-configuration2-2.1.1.jar:2.1.1]
> at
> org.apache.commons.configuration2.beanutils.BeanHelper.initProperty(BeanHelper.java:357)
> [commons-configuration2-2.1.1.jar:2.1.1]
> at
> org.apache.commons.configuration2.beanutils.BeanHelper.initBeanProperties(BeanHelper.java:273)
> [commons-configuration2-2.1.1.jar:2.1.1]
> at
> org.apache.commons.configuration2.beanutils.BeanHelper.initBean(BeanHelper.java:192)
> [commons-configuration2-2.1.1.jar:2.1.1]
[jira] [Updated] (HIVE-26865) Fix TestSQL11ReservedKeyWordsNegative test in branch-3
[ https://issues.apache.org/jira/browse/HIVE-26865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Raj updated HIVE-26865: Fix Version/s: 3.2.0 > Fix TestSQL11ReservedKeyWordsNegative test in branch-3 > -- > > Key: HIVE-26865 > URL: https://issues.apache.org/jira/browse/HIVE-26865 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: László Bodor >Priority: Critical > Labels: pull-request-available > Fix For: 3.2.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Due to [HIVE-21293] Fix ambiguity in grammar warnings at compilation time > (II) - ASF JIRA (apache.org), the test cases are failing with the following > error : > java.lang.AssertionError: Expected ParseException > at org.junit.Assert.fail(Assert.java:88) > at > org.apache.hadoop.hive.ql.parse.TestSQL11ReservedKeyWordsNegative$TestSQL11ReservedKeyWordsNegativeParametrized.testNegative(TestSQL11ReservedKeyWordsNegative.java:105) > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.junit.runners.Suite.runChild(Suite.java:127) > at org.junit.runners.Suite.runChild(Suite.java:26) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.junit.runners.Suite.runChild(Suite.java:127) > at org.junit.runners.Suite.runChild(Suite.java:26) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340) > at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27154) Fix testBootstrapReplLoadRetryAfterFailureForPartitions
[ https://issues.apache.org/jira/browse/HIVE-27154?focusedWorklogId=852913=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852913 ] ASF GitHub Bot logged work on HIVE-27154: - Author: ASF GitHub Bot Created on: 24/Mar/23 19:07 Start Date: 24/Mar/23 19:07 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #4149: URL: https://github.com/apache/hive/pull/4149#issuecomment-1483285514 Already fixed using #4148 #4151 Issue Time Tracking --- Worklog Id: (was: 852913) Time Spent: 1h (was: 50m) > Fix testBootstrapReplLoadRetryAfterFailureForPartitions > --- > > Key: HIVE-27154 > URL: https://issues.apache.org/jira/browse/HIVE-27154 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > `testBootstrapReplLoadRetryAfterFailureForPartitions` has been failing on > branch-3 > > http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27154) Fix testBootstrapReplLoadRetryAfterFailureForPartitions
[ https://issues.apache.org/jira/browse/HIVE-27154?focusedWorklogId=852910=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852910 ] ASF GitHub Bot logged work on HIVE-27154: - Author: ASF GitHub Bot Created on: 24/Mar/23 19:06 Start Date: 24/Mar/23 19:06 Worklog Time Spent: 10m Work Description: amanraj2520 closed pull request #4149: HIVE-27154 : Backport of HIVE-20953 : Remove a function from function registry when it can not be added to the metastore when creating it. URL: https://github.com/apache/hive/pull/4149 Issue Time Tracking --- Worklog Id: (was: 852910) Time Spent: 50m (was: 40m) > Fix testBootstrapReplLoadRetryAfterFailureForPartitions > --- > > Key: HIVE-27154 > URL: https://issues.apache.org/jira/browse/HIVE-27154 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > `testBootstrapReplLoadRetryAfterFailureForPartitions` has been failing on > branch-3 > > http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27154) Fix testBootstrapReplLoadRetryAfterFailureForPartitions
[ https://issues.apache.org/jira/browse/HIVE-27154?focusedWorklogId=852909=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852909 ] ASF GitHub Bot logged work on HIVE-27154: - Author: ASF GitHub Bot Created on: 24/Mar/23 19:05 Start Date: 24/Mar/23 19:05 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #4151: URL: https://github.com/apache/hive/pull/4151#issuecomment-1483283803 Big +1 LGTM. This is a clean backport and necessary for the test fix. Issue Time Tracking --- Worklog Id: (was: 852909) Time Spent: 40m (was: 0.5h) > Fix testBootstrapReplLoadRetryAfterFailureForPartitions > --- > > Key: HIVE-27154 > URL: https://issues.apache.org/jira/browse/HIVE-27154 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > `testBootstrapReplLoadRetryAfterFailureForPartitions` has been failing on > branch-3 > > http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27154) Fix testBootstrapReplLoadRetryAfterFailureForPartitions
[ https://issues.apache.org/jira/browse/HIVE-27154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17704762#comment-17704762 ] Vihang Karajgaonkar commented on HIVE-27154: This was hard to reproduce. The problem was occurs when the DatabaseEventsIterator returns tables fileStatus before functions. This was fixed in master branch in HIVE-20953. HIVE-20953 also depends on HIVE-20680 so we should backport that too. > Fix testBootstrapReplLoadRetryAfterFailureForPartitions > --- > > Key: HIVE-27154 > URL: https://issues.apache.org/jira/browse/HIVE-27154 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > `testBootstrapReplLoadRetryAfterFailureForPartitions` has been failing on > branch-3 > > http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27154) Fix testBootstrapReplLoadRetryAfterFailureForPartitions
[ https://issues.apache.org/jira/browse/HIVE-27154?focusedWorklogId=852906=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852906 ] ASF GitHub Bot logged work on HIVE-27154: - Author: ASF GitHub Bot Created on: 24/Mar/23 18:55 Start Date: 24/Mar/23 18:55 Worklog Time Spent: 10m Work Description: vihangk1 commented on PR #4151: URL: https://github.com/apache/hive/pull/4151#issuecomment-1483271665 @amanraj2520 Can you please review this? Thanks! Issue Time Tracking --- Worklog Id: (was: 852906) Time Spent: 0.5h (was: 20m) > Fix testBootstrapReplLoadRetryAfterFailureForPartitions > --- > > Key: HIVE-27154 > URL: https://issues.apache.org/jira/browse/HIVE-27154 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > `testBootstrapReplLoadRetryAfterFailureForPartitions` has been failing on > branch-3 > > http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27154) Fix testBootstrapReplLoadRetryAfterFailureForPartitions
[ https://issues.apache.org/jira/browse/HIVE-27154?focusedWorklogId=852905=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852905 ] ASF GitHub Bot logged work on HIVE-27154: - Author: ASF GitHub Bot Created on: 24/Mar/23 18:55 Start Date: 24/Mar/23 18:55 Worklog Time Spent: 10m Work Description: vihangk1 opened a new pull request, #4151: URL: https://github.com/apache/hive/pull/4151 ### What changes were proposed in this pull request? This PR backports HIVE-20953 which is needed to fix testBootstrapReplLoadRetryAfterFailureForPartitions ### Why are the changes needed? Test fix ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? CI Job Issue Time Tracking --- Worklog Id: (was: 852905) Time Spent: 20m (was: 10m) > Fix testBootstrapReplLoadRetryAfterFailureForPartitions > --- > > Key: HIVE-27154 > URL: https://issues.apache.org/jira/browse/HIVE-27154 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > `testBootstrapReplLoadRetryAfterFailureForPartitions` has been failing on > branch-3 > > http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27171) Backport HIVE-20680 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar resolved HIVE-27171. Fix Version/s: 3.2.0 Resolution: Fixed > Backport HIVE-20680 to branch-3 > --- > > Key: HIVE-27171 > URL: https://issues.apache.org/jira/browse/HIVE-27171 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > We need to backport HIVE-26836 to fix the > TestReplicationScenariosAcrossInstances on branch-3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-27171) Backport HIVE-20680 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-27171 started by Vihang Karajgaonkar. -- > Backport HIVE-20680 to branch-3 > --- > > Key: HIVE-27171 > URL: https://issues.apache.org/jira/browse/HIVE-27171 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > We need to backport HIVE-26836 to fix the > TestReplicationScenariosAcrossInstances on branch-3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27171) Backport HIVE-20680 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27171?focusedWorklogId=852904=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852904 ] ASF GitHub Bot logged work on HIVE-27171: - Author: ASF GitHub Bot Created on: 24/Mar/23 18:51 Start Date: 24/Mar/23 18:51 Worklog Time Spent: 10m Work Description: vihangk1 commented on PR #4148: URL: https://github.com/apache/hive/pull/4148#issuecomment-1483266050 Thanks @amanraj2520. Merged. Issue Time Tracking --- Worklog Id: (was: 852904) Time Spent: 1h 10m (was: 1h) > Backport HIVE-20680 to branch-3 > --- > > Key: HIVE-27171 > URL: https://issues.apache.org/jira/browse/HIVE-27171 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > We need to backport HIVE-26836 to fix the > TestReplicationScenariosAcrossInstances on branch-3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27171) Backport HIVE-20680 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27171?focusedWorklogId=852903=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852903 ] ASF GitHub Bot logged work on HIVE-27171: - Author: ASF GitHub Bot Created on: 24/Mar/23 18:50 Start Date: 24/Mar/23 18:50 Worklog Time Spent: 10m Work Description: vihangk1 merged PR #4148: URL: https://github.com/apache/hive/pull/4148 Issue Time Tracking --- Worklog Id: (was: 852903) Time Spent: 1h (was: 50m) > Backport HIVE-20680 to branch-3 > --- > > Key: HIVE-27171 > URL: https://issues.apache.org/jira/browse/HIVE-27171 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > We need to backport HIVE-26836 to fix the > TestReplicationScenariosAcrossInstances on branch-3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27172) Add the HMS client connection timeout config
[
https://issues.apache.org/jira/browse/HIVE-27172?focusedWorklogId=852901=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852901
]
ASF GitHub Bot logged work on HIVE-27172:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 18:39
Start Date: 24/Mar/23 18:39
Worklog Time Spent: 10m
Work Description: pan3793 commented on code in PR #4150:
URL: https://github.com/apache/hive/pull/4150#discussion_r1147940693
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##
@@ -402,6 +402,8 @@ public enum ConfVars {
"has an infinite lifetime."),
CLIENT_SOCKET_TIMEOUT("metastore.client.socket.timeout",
"hive.metastore.client.socket.timeout", 600,
TimeUnit.SECONDS, "MetaStore Client socket timeout in seconds"),
+CLIENT_CONNECTION_TIMEOUT("metastore.client.connection.timeout",
"hive.metastore.client.connection.timeout", 10,
Review Comment:
It makes sense to set connection timeout a small value, because generally,
establish the connection is quick, and small timeout make it quick failover if
one HMS is down, but it silent changes the default behavior, I'm not sure if
such change is acceptable in Hive community
Issue Time Tracking
---
Worklog Id: (was: 852901)
Time Spent: 40m (was: 0.5h)
> Add the HMS client connection timeout config
>
>
> Key: HIVE-27172
> URL: https://issues.apache.org/jira/browse/HIVE-27172
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently {{HiveMetastoreClient}} use {{CLIENT_SOCKET_TIMEOUT}} as both
> socket timeout and connection timeout, it's not convenient for users to set a
> smaller connection timeout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27113) Increasing default for hive.thrift.client.max.message.size to 2 GB
[
https://issues.apache.org/jira/browse/HIVE-27113?focusedWorklogId=852900=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852900
]
ASF GitHub Bot logged work on HIVE-27113:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 18:38
Start Date: 24/Mar/23 18:38
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #4137:
URL: https://github.com/apache/hive/pull/4137#issuecomment-1483252549
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4137)
[](https://sonarcloud.io/project/issues?id=apache_hive=4137=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=4137=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4137=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=4137=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=4137=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4137=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4137=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4137=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4137=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=4137=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=4137=false=CODE_SMELL)
[2 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=4137=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=4137=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=4137=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 852900)
Time Spent: 1.5h (was: 1h 20m)
> Increasing default for hive.thrift.client.max.message.size to 2 GB
> --
>
> Key: HIVE-27113
> URL: https://issues.apache.org/jira/browse/HIVE-27113
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> HIVE_THRIFT_CLIENT_MAX_MESSAGE_SIZE("hive.thrift.client.max.message.size",
> "1gb",
> new SizeValidator(-1L, true, (long) Integer.MAX_VALUE, true),
> "Thrift client configuration for max message size. 0 or -1 will use
> the default defined in the Thrift " +
> "library. The upper limit is 2147483648 bytes (or 2gb).")
> Documentation on the help suggests setting 2147483648 while Integer Max is
> 2147483647. So, it actually becomes -1 and gets set to thrift default limit
> (100 MB)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27172) Add the HMS client connection timeout config
[
https://issues.apache.org/jira/browse/HIVE-27172?focusedWorklogId=852898=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852898
]
ASF GitHub Bot logged work on HIVE-27172:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 18:30
Start Date: 24/Mar/23 18:30
Worklog Time Spent: 10m
Work Description: pan3793 commented on PR #4150:
URL: https://github.com/apache/hive/pull/4150#issuecomment-1483243678
A similar PR https://github.com/apache/hive/pull/3379 for JDBC thrift
client, IMO the connection timeout is quite important, please help reopen the
closed PR if any committer think it's useful.
Issue Time Tracking
---
Worklog Id: (was: 852898)
Time Spent: 0.5h (was: 20m)
> Add the HMS client connection timeout config
>
>
> Key: HIVE-27172
> URL: https://issues.apache.org/jira/browse/HIVE-27172
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently {{HiveMetastoreClient}} use {{CLIENT_SOCKET_TIMEOUT}} as both
> socket timeout and connection timeout, it's not convenient for users to set a
> smaller connection timeout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27172) Add the HMS client connection timeout config
[
https://issues.apache.org/jira/browse/HIVE-27172?focusedWorklogId=852896=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852896
]
ASF GitHub Bot logged work on HIVE-27172:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 18:24
Start Date: 24/Mar/23 18:24
Worklog Time Spent: 10m
Work Description: wecharyu commented on code in PR #4150:
URL: https://github.com/apache/hive/pull/4150#discussion_r1147927316
##
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreTimeout.java:
##
@@ -43,9 +46,10 @@ public class TestHiveMetaStoreTimeout {
protected static HiveMetaStoreClient client;
protected static Configuration conf;
protected static Warehouse warehouse;
+ protected static int port;
- @BeforeClass
- public static void setUp() throws Exception {
+ @Before
+ public void setUp() throws Exception {
HMSHandler.testTimeoutEnabled = true;
Review Comment:
Nit: This test code hacks `HMSHandler` class, I will refactor this in a new
PR.
Issue Time Tracking
---
Worklog Id: (was: 852896)
Time Spent: 20m (was: 10m)
> Add the HMS client connection timeout config
>
>
> Key: HIVE-27172
> URL: https://issues.apache.org/jira/browse/HIVE-27172
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently {{HiveMetastoreClient}} use {{CLIENT_SOCKET_TIMEOUT}} as both
> socket timeout and connection timeout, it's not convenient for users to set a
> smaller connection timeout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27172) Add the HMS client connection timeout config
[
https://issues.apache.org/jira/browse/HIVE-27172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-27172:
--
Labels: pull-request-available (was: )
> Add the HMS client connection timeout config
>
>
> Key: HIVE-27172
> URL: https://issues.apache.org/jira/browse/HIVE-27172
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently {{HiveMetastoreClient}} use {{CLIENT_SOCKET_TIMEOUT}} as both
> socket timeout and connection timeout, it's not convenient for users to set a
> smaller connection timeout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27172) Add the HMS client connection timeout config
[
https://issues.apache.org/jira/browse/HIVE-27172?focusedWorklogId=852892=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852892
]
ASF GitHub Bot logged work on HIVE-27172:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 18:16
Start Date: 24/Mar/23 18:16
Worklog Time Spent: 10m
Work Description: wecharyu opened a new pull request, #4150:
URL: https://github.com/apache/hive/pull/4150
### What changes were proposed in this pull request?
Currently `HiveMetaStoreClient` use `CLIENT_SOCKET_TIMEOUT` as both socket
timeout and connection timeout, we want to introduce a new HiveMetaStoreClient
config for connection timeout:
```java
CLIENT_CONNECTION_TIMEOUT("metastore.client.connection.timeout",
"hive.metastore.client.connection.timeout", 10,
TimeUnit.SECONDS, "MetaStore Client connection timeout in
seconds"),
```
### Why are the changes needed?
1. achieve a more flexible config for client socket timeout and connection
timeout.
2. we can use a smaller connection timeout to fail-fast reconnect to another
HMS server.
### Does this PR introduce _any_ user-facing change?
Yes, user can set the client connection timeout by this conf.
### How was this patch tested?
Add a unit test.
Issue Time Tracking
---
Worklog Id: (was: 852892)
Remaining Estimate: 0h
Time Spent: 10m
> Add the HMS client connection timeout config
>
>
> Key: HIVE-27172
> URL: https://issues.apache.org/jira/browse/HIVE-27172
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently {{HiveMetastoreClient}} use {{CLIENT_SOCKET_TIMEOUT}} as both
> socket timeout and connection timeout, it's not convenient for users to set a
> smaller connection timeout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27172) Add the HMS client connection timeout config
[
https://issues.apache.org/jira/browse/HIVE-27172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-27172:
--
Issue Type: Task (was: New Feature)
> Add the HMS client connection timeout config
>
>
> Key: HIVE-27172
> URL: https://issues.apache.org/jira/browse/HIVE-27172
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> Currently {{HiveMetastoreClient}} use {{CLIENT_SOCKET_TIMEOUT}} as both
> socket timeout and connection timeout, it's not convenient for users to set a
> smaller connection timeout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27172) Add the HMS client connection timeout config
[
https://issues.apache.org/jira/browse/HIVE-27172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar reassigned HIVE-27172:
-
> Add the HMS client connection timeout config
>
>
> Key: HIVE-27172
> URL: https://issues.apache.org/jira/browse/HIVE-27172
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> Currently {{HiveMetastoreClient}} use {{CLIENT_SOCKET_TIMEOUT}} as both
> socket timeout and connection timeout, it's not convenient for users to set a
> smaller connection timeout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27113) Increasing default for hive.thrift.client.max.message.size to 2 GB
[
https://issues.apache.org/jira/browse/HIVE-27113?focusedWorklogId=852882=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852882
]
ASF GitHub Bot logged work on HIVE-27113:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 17:19
Start Date: 24/Mar/23 17:19
Worklog Time Spent: 10m
Work Description: jfsii commented on code in PR #4137:
URL: https://github.com/apache/hive/pull/4137#discussion_r1147867743
##
service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java:
##
@@ -310,7 +310,7 @@ protected synchronized TTransport connect(HiveConf conf)
throws HiveSQLException
String host = conf.getVar(HiveConf.ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST);
int port = conf.getIntVar(HiveConf.ConfVars.HIVE_SERVER2_THRIFT_PORT);
-int maxThriftMessageSize = (int)
conf.getSizeVar(HiveConf.ConfVars.HIVE_THRIFT_CLIENT_MAX_MESSAGE_SIZE);
+int maxThriftMessageSize = (int)
Math.min(conf.getSizeVar(HiveConf.ConfVars.HIVE_THRIFT_CLIENT_MAX_MESSAGE_SIZE),Integer.MAX_VALUE);
Review Comment:
I'm not sure the sizeValidator gets executed when a hive-site.xml is loaded
(only when a SET is called), I'm not 100% sure though. That is the reason for
my paranoia. Also having a helper method for that does the min and max range
would make it usable in other instances in which we need to convert a "size" to
an int or some other range.
It is not an absolute requirement, mostly a suggestion - if
@saihemanth-cloudera is okay with the state of the patch that is good enough
with me. (Functionally it is correct which is what matters).
Issue Time Tracking
---
Worklog Id: (was: 852882)
Time Spent: 1h 20m (was: 1h 10m)
> Increasing default for hive.thrift.client.max.message.size to 2 GB
> --
>
> Key: HIVE-27113
> URL: https://issues.apache.org/jira/browse/HIVE-27113
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> HIVE_THRIFT_CLIENT_MAX_MESSAGE_SIZE("hive.thrift.client.max.message.size",
> "1gb",
> new SizeValidator(-1L, true, (long) Integer.MAX_VALUE, true),
> "Thrift client configuration for max message size. 0 or -1 will use
> the default defined in the Thrift " +
> "library. The upper limit is 2147483648 bytes (or 2gb).")
> Documentation on the help suggests setting 2147483648 while Integer Max is
> 2147483647. So, it actually becomes -1 and gets set to thrift default limit
> (100 MB)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27157) AssertionError when inferring return type for unix_timestamp function
[
https://issues.apache.org/jira/browse/HIVE-27157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis resolved HIVE-27157.
Fix Version/s: 4.0.0
Resolution: Fixed
Fixed in
https://github.com/apache/hive/commit/b30fe4c1af9d134e7e533a715f2401a67530d016.
Thanks for the review [~kkasa]!
> AssertionError when inferring return type for unix_timestamp function
> -
>
> Key: HIVE-27157
> URL: https://issues.apache.org/jira/browse/HIVE-27157
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0-alpha-2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Any attempt to derive the return data type for the {{unix_timestamp}}
> function results into the following assertion error.
> {noformat}
> java.lang.AssertionError: typeName.allowsPrecScale(true, false): BIGINT
> at
> org.apache.calcite.sql.type.BasicSqlType.checkPrecScale(BasicSqlType.java:65)
> at org.apache.calcite.sql.type.BasicSqlType.(BasicSqlType.java:81)
> at
> org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:67)
> at
> org.apache.calcite.sql.fun.SqlAbstractTimeFunction.inferReturnType(SqlAbstractTimeFunction.java:78)
> at
> org.apache.calcite.rex.RexBuilder.deriveReturnType(RexBuilder.java:278)
> {noformat}
> due to a faulty implementation of type inference for the respective operators:
> *
> [https://github.com/apache/hive/blob/52360151dc43904217e812efde1069d6225e9570/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnixTimestampSqlOperator.java]
> *
> [https://github.com/apache/hive/blob/52360151dc43904217e812efde1069d6225e9570/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveToUnixTimestampSqlOperator.java]
> Although at this stage in master it is not possible to reproduce the problem
> with an actual SQL query the buggy implementation must be fixed since slight
> changes in the code/CBO rules may lead to methods relying on
> {{{}SqlOperator.inferReturnType{}}}.
> Note that in older versions of Hive it is possible to hit the AssertionError
> in various ways. For example in Hive 3.1.3 (and older), the error may come
> from
> [HiveRelDecorrelator|https://github.com/apache/hive/blob/4df4d75bf1e16fe0af75aad0b4179c34c07fc975/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java#L1933]
> in the presence of sub-queries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27157) AssertionError when inferring return type for unix_timestamp function
[
https://issues.apache.org/jira/browse/HIVE-27157?focusedWorklogId=852878=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852878
]
ASF GitHub Bot logged work on HIVE-27157:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 16:28
Start Date: 24/Mar/23 16:28
Worklog Time Spent: 10m
Work Description: zabetak closed pull request #4135: HIVE-27157:
AssertionError when inferring return type for unix_timestamp function
URL: https://github.com/apache/hive/pull/4135
Issue Time Tracking
---
Worklog Id: (was: 852878)
Time Spent: 1.5h (was: 1h 20m)
> AssertionError when inferring return type for unix_timestamp function
> -
>
> Key: HIVE-27157
> URL: https://issues.apache.org/jira/browse/HIVE-27157
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0-alpha-2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Any attempt to derive the return data type for the {{unix_timestamp}}
> function results into the following assertion error.
> {noformat}
> java.lang.AssertionError: typeName.allowsPrecScale(true, false): BIGINT
> at
> org.apache.calcite.sql.type.BasicSqlType.checkPrecScale(BasicSqlType.java:65)
> at org.apache.calcite.sql.type.BasicSqlType.(BasicSqlType.java:81)
> at
> org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:67)
> at
> org.apache.calcite.sql.fun.SqlAbstractTimeFunction.inferReturnType(SqlAbstractTimeFunction.java:78)
> at
> org.apache.calcite.rex.RexBuilder.deriveReturnType(RexBuilder.java:278)
> {noformat}
> due to a faulty implementation of type inference for the respective operators:
> *
> [https://github.com/apache/hive/blob/52360151dc43904217e812efde1069d6225e9570/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnixTimestampSqlOperator.java]
> *
> [https://github.com/apache/hive/blob/52360151dc43904217e812efde1069d6225e9570/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveToUnixTimestampSqlOperator.java]
> Although at this stage in master it is not possible to reproduce the problem
> with an actual SQL query the buggy implementation must be fixed since slight
> changes in the code/CBO rules may lead to methods relying on
> {{{}SqlOperator.inferReturnType{}}}.
> Note that in older versions of Hive it is possible to hit the AssertionError
> in various ways. For example in Hive 3.1.3 (and older), the error may come
> from
> [HiveRelDecorrelator|https://github.com/apache/hive/blob/4df4d75bf1e16fe0af75aad0b4179c34c07fc975/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java#L1933]
> in the presence of sub-queries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[ https://issues.apache.org/jira/browse/HIVE-27168?focusedWorklogId=852877=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852877 ] ASF GitHub Bot logged work on HIVE-27168: - Author: ASF GitHub Bot Created on: 24/Mar/23 16:22 Start Date: 24/Mar/23 16:22 Worklog Time Spent: 10m Work Description: veghlaci05 merged PR #4144: URL: https://github.com/apache/hive/pull/4144 Issue Time Tracking --- Worklog Id: (was: 852877) Time Spent: 3h (was: 2h 50m) > Use basename of the datatype when fetching partition metadata using partition > filters > - > > Key: HIVE-27168 > URL: https://issues.apache.org/jira/browse/HIVE-27168 > Project: Hive > Issue Type: Bug >Reporter: Sourabh Badhya >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > While fetching partition metadata using partition filters, we use the column > type of the table directly. However, char/varchar types can contain extra > information such as length of the char/varchar column and hence it skips > fetching partition metadata due to this extra information. > Solution: Use the basename of the column type while deciding on whether > partition pruning can be done on the partitioned column. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[ https://issues.apache.org/jira/browse/HIVE-27168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Végh updated HIVE-27168: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Merged to master, [~sbadhya] thanks for the patch! > Use basename of the datatype when fetching partition metadata using partition > filters > - > > Key: HIVE-27168 > URL: https://issues.apache.org/jira/browse/HIVE-27168 > Project: Hive > Issue Type: Bug >Reporter: Sourabh Badhya >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 3h > Remaining Estimate: 0h > > While fetching partition metadata using partition filters, we use the column > type of the table directly. However, char/varchar types can contain extra > information such as length of the char/varchar column and hence it skips > fetching partition metadata due to this extra information. > Solution: Use the basename of the column type while deciding on whether > partition pruning can be done on the partitioned column. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27171) Backport HIVE-20680 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27171?focusedWorklogId=852876=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852876 ] ASF GitHub Bot logged work on HIVE-27171: - Author: ASF GitHub Bot Created on: 24/Mar/23 15:59 Start Date: 24/Mar/23 15:59 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #4148: URL: https://github.com/apache/hive/pull/4148#issuecomment-1483043453 @vihangk1 This is the exact and accurate fix I had in mind for this test. +1 from my side. Once we cherry pick FunctionTask changes, we can merge this Issue Time Tracking --- Worklog Id: (was: 852876) Time Spent: 50m (was: 40m) > Backport HIVE-20680 to branch-3 > --- > > Key: HIVE-27171 > URL: https://issues.apache.org/jira/browse/HIVE-27171 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > We need to backport HIVE-26836 to fix the > TestReplicationScenariosAcrossInstances on branch-3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27171) Backport HIVE-20680 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27171?focusedWorklogId=852875=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852875 ] ASF GitHub Bot logged work on HIVE-27171: - Author: ASF GitHub Bot Created on: 24/Mar/23 15:54 Start Date: 24/Mar/23 15:54 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #4148: URL: https://github.com/apache/hive/pull/4148#issuecomment-1483035277 I see commits in FunctionTask missing https://github.com/apache/hive/pull/4147/files#diff-bf63347dcc25dc9cec3cbe9288dde549df8ac3054a5d8e0e33bdc930d013907e in this PR Issue Time Tracking --- Worklog Id: (was: 852875) Time Spent: 40m (was: 0.5h) > Backport HIVE-20680 to branch-3 > --- > > Key: HIVE-27171 > URL: https://issues.apache.org/jira/browse/HIVE-27171 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > We need to backport HIVE-26836 to fix the > TestReplicationScenariosAcrossInstances on branch-3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27171) Backport HIVE-20680 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27171?focusedWorklogId=852874=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852874 ] ASF GitHub Bot logged work on HIVE-27171: - Author: ASF GitHub Bot Created on: 24/Mar/23 15:52 Start Date: 24/Mar/23 15:52 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #4148: URL: https://github.com/apache/hive/pull/4148#issuecomment-1483030399 @vihangk1 I see this test still failing on this PR. Any difference between #4147 and this? Issue Time Tracking --- Worklog Id: (was: 852874) Time Spent: 0.5h (was: 20m) > Backport HIVE-20680 to branch-3 > --- > > Key: HIVE-27171 > URL: https://issues.apache.org/jira/browse/HIVE-27171 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > We need to backport HIVE-26836 to fix the > TestReplicationScenariosAcrossInstances on branch-3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27171) Backport HIVE-20680 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27171?focusedWorklogId=852873=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852873 ] ASF GitHub Bot logged work on HIVE-27171: - Author: ASF GitHub Bot Created on: 24/Mar/23 15:48 Start Date: 24/Mar/23 15:48 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #4148: URL: https://github.com/apache/hive/pull/4148#issuecomment-1483024608 @vihangk1 What a coincidence!! I identified the same tickets and raised the PR at the same time as you. PR for reference - https://github.com/apache/hive/pull/4149 Issue Time Tracking --- Worklog Id: (was: 852873) Time Spent: 20m (was: 10m) > Backport HIVE-20680 to branch-3 > --- > > Key: HIVE-27171 > URL: https://issues.apache.org/jira/browse/HIVE-27171 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > We need to backport HIVE-26836 to fix the > TestReplicationScenariosAcrossInstances on branch-3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27154) Fix testBootstrapReplLoadRetryAfterFailureForPartitions
[ https://issues.apache.org/jira/browse/HIVE-27154?focusedWorklogId=852872=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852872 ] ASF GitHub Bot logged work on HIVE-27154: - Author: ASF GitHub Bot Created on: 24/Mar/23 15:44 Start Date: 24/Mar/23 15:44 Worklog Time Spent: 10m Work Description: amanraj2520 opened a new pull request, #4149: URL: https://github.com/apache/hive/pull/4149 JIRA link : https://issues.apache.org/jira/browse/HIVE-27154 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Issue Time Tracking --- Worklog Id: (was: 852872) Remaining Estimate: 0h Time Spent: 10m > Fix testBootstrapReplLoadRetryAfterFailureForPartitions > --- > > Key: HIVE-27154 > URL: https://issues.apache.org/jira/browse/HIVE-27154 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > `testBootstrapReplLoadRetryAfterFailureForPartitions` has been failing on > branch-3 > > http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27154) Fix testBootstrapReplLoadRetryAfterFailureForPartitions
[ https://issues.apache.org/jira/browse/HIVE-27154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27154: -- Labels: pull-request-available (was: ) > Fix testBootstrapReplLoadRetryAfterFailureForPartitions > --- > > Key: HIVE-27154 > URL: https://issues.apache.org/jira/browse/HIVE-27154 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > `testBootstrapReplLoadRetryAfterFailureForPartitions` has been failing on > branch-3 > > http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27150) Drop single partition can also support direct sql
[ https://issues.apache.org/jira/browse/HIVE-27150?focusedWorklogId=852871=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852871 ] ASF GitHub Bot logged work on HIVE-27150: - Author: ASF GitHub Bot Created on: 24/Mar/23 15:43 Start Date: 24/Mar/23 15:43 Worklog Time Spent: 10m Work Description: wecharyu commented on PR #4123: URL: https://github.com/apache/hive/pull/4123#issuecomment-1483018000 @deniskuzZ @kasakrisz @saihemanth-cloudera: Could you please review this PR? Issue Time Tracking --- Worklog Id: (was: 852871) Time Spent: 0.5h (was: 20m) > Drop single partition can also support direct sql > - > > Key: HIVE-27150 > URL: https://issues.apache.org/jira/browse/HIVE-27150 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > *Background:* > [HIVE-6980|https://issues.apache.org/jira/browse/HIVE-6980] supports direct > sql for drop_partitions, we can reuse this huge improvement in drop_partition. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27171) Backport HIVE-20680 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27171?focusedWorklogId=852870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852870 ] ASF GitHub Bot logged work on HIVE-27171: - Author: ASF GitHub Bot Created on: 24/Mar/23 15:31 Start Date: 24/Mar/23 15:31 Worklog Time Spent: 10m Work Description: vihangk1 commented on PR #4148: URL: https://github.com/apache/hive/pull/4148#issuecomment-1483000294 @amanraj2520 Can you please review this. We need to backport HIVE-20680 and HIVE-20953 to fix the TestReplicationScenariosAcrossInstances on branch-3. I tested these backports fix the test in https://github.com/apache/hive/pull/4147 Issue Time Tracking --- Worklog Id: (was: 852870) Remaining Estimate: 0h Time Spent: 10m > Backport HIVE-20680 to branch-3 > --- > > Key: HIVE-27171 > URL: https://issues.apache.org/jira/browse/HIVE-27171 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > We need to backport HIVE-26836 to fix the > TestReplicationScenariosAcrossInstances on branch-3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27171) Backport HIVE-20680 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27171: -- Labels: pull-request-available (was: ) > Backport HIVE-20680 to branch-3 > --- > > Key: HIVE-27171 > URL: https://issues.apache.org/jira/browse/HIVE-27171 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > We need to backport HIVE-26836 to fix the > TestReplicationScenariosAcrossInstances on branch-3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27171) Backport HIVE-20680 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-27171: -- > Backport HIVE-20680 to branch-3 > --- > > Key: HIVE-27171 > URL: https://issues.apache.org/jira/browse/HIVE-27171 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > > We need to backport HIVE-26836 to fix the > TestReplicationScenariosAcrossInstances on branch-3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[ https://issues.apache.org/jira/browse/HIVE-27168?focusedWorklogId=852859=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852859 ] ASF GitHub Bot logged work on HIVE-27168: - Author: ASF GitHub Bot Created on: 24/Mar/23 14:01 Start Date: 24/Mar/23 14:01 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4144: URL: https://github.com/apache/hive/pull/4144#issuecomment-1482851818 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4144) [](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4144=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4144=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4144=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4144=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4144=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 852859) Time Spent: 2h 50m (was: 2h 40m) > Use basename of the datatype when fetching partition metadata using partition > filters > - > > Key: HIVE-27168 > URL: https://issues.apache.org/jira/browse/HIVE-27168 > Project: Hive > Issue Type: Bug >Reporter: Sourabh Badhya >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > While fetching partition metadata using partition filters, we use the column > type of the table directly. However, char/varchar types can contain extra > information such as length of the char/varchar column and hence it skips > fetching partition metadata due to this extra information. > Solution: Use the basename of the column type while deciding on whether > partition pruning can be done on the partitioned column. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26674) REBALANCE type compaction
[
https://issues.apache.org/jira/browse/HIVE-26674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Végh resolved HIVE-26674.
Fix Version/s: 4.0.0
Target Version/s: 4.0.0
Resolution: Fixed
> REBALANCE type compaction
> -
>
> Key: HIVE-26674
> URL: https://issues.apache.org/jira/browse/HIVE-26674
> Project: Hive
> Issue Type: Improvement
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
> Labels: compaction
> Fix For: 4.0.0
>
>
> h2. Problem statement:
> Without explicit bucketing defined, bucket files are very sensitive to the
> amount of data loaded/modified in the table.
> When
> * there are initial or larger time-window loads or reloads beside smaller
> load schedules (like initial and monthly vs. daily loads)
> * or even if load scheduling is periodic but the volume of the data changes
> are not,
> * or even if data volume and periodicity are all balanced but runtime
> resources affect the loader application to run on different number of tasks
> The data loaded into non-explicitly bucketed full-acid ORC tables can lead to
> unbalanced bucketed tables over time!
> The number of buckets is calculated from the amount of data to be loaded. If
> the table is created with a huge amount of initial data (which will create
> several buckets), and then only a few records are added to it (which will be
> written only into the first 1-2 buckets), but frequently, the result will be
> that the data is unbalanced within the buckets. The first few buckets will
> contain much more data than the others.
> h2. Concept:
> h4. Rebalancing compaction
> A new compaction type (‘REBALANCE’) should be created to address the issue
> for badly balanced data among buckets. This compaction type would result in a
> table like an INSERT-OVERWRITE would lead to. New base and independent bucket
> indexes from the previous base or deltas. The new number of buckets can be
> optionally supplied, otherwise the new table would still have the same amount
> of buckets, but with re-balanced data.
> h4. Sorting
> Optionally, a sorting expression can be supplied, to be able to re-sort the
> data during the rebalance.
> The expression can be supplied in two ways:
> * Via the ALTER TABLE COMPACT:
> ALTER TABLE COMPACT ‘REBALANCE’ ORDER BY ASC|DESC
> h4. Manual rebalance
> The rebalance request can be created by using the ALTER TABLE COMPACT command
> (E.g. manual compaction).
> h4. Limitations
> * Rebalancing can be done only within partitions.
> * Rebalancing is not possible on explicitly bucketed (clustered) tables
> * Rebalancing is not possible via MR based compaction
> * Rebalancing is not supported on insert-only tables
> h2. Implications
> h4. Compaction request (DB schema) changes
> * A new compaction type (REBALANCE) must be added to the allowed compaction
> TYPES.
> * A new optional field (and nullable DB column) is required to store the
> number of requested implicit buckets.
> h4. ALTER TABLE COMPACT changes
> The ALTER TABLE COMPACT command must accept the
> * ‘REBALANCE’, compaction type
> * optionally the new number of the required buckets (... INTO \{N} BUCKETS).
> * Optionally the sorting expression (ORDER BY column ASC, columnB DESC)
> h4. Compactor changes
> Both the MR and query based compaction tasks must be enhanced with the
> ability to do a rebalancing compaction.
> h4. Query based compaction changes
> New compactor implementations are required:
> * Query based rebalance compactor for fully acid tables
> h4. MR based compaction changes
> MR is deprecated, rebalancing compaction will only be implemented, if it’s
> really easy to do so.
> h2. Open points
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27158) Store hive columns stats in puffin files for iceberg tables
[ https://issues.apache.org/jira/browse/HIVE-27158?focusedWorklogId=852849=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852849 ] ASF GitHub Bot logged work on HIVE-27158: - Author: ASF GitHub Bot Created on: 24/Mar/23 13:19 Start Date: 24/Mar/23 13:19 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4131: URL: https://github.com/apache/hive/pull/4131#issuecomment-1482786217 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4131) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4131=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4131=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4131=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=CODE_SMELL) [6 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4131=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4131=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 852849) Time Spent: 1h 10m (was: 1h) > Store hive columns stats in puffin files for iceberg tables > --- > > Key: HIVE-27158 > URL: https://issues.apache.org/jira/browse/HIVE-27158 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[
https://issues.apache.org/jira/browse/HIVE-27168?focusedWorklogId=852848=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852848
]
ASF GitHub Bot logged work on HIVE-27168:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 12:56
Start Date: 24/Mar/23 12:56
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #4144:
URL: https://github.com/apache/hive/pull/4144#discussion_r1147527514
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##
@@ -1327,7 +1327,7 @@ public void visit(LeafNode node) throws MetaException {
return;
}
- String colTypeStr = partitionKeys.get(partColIndex).getType();
+ String colTypeStr =
MetaStoreUtils.getBaseNameOfColType(partitionKeys.get(partColIndex).getType());
Review Comment:
Yes you are right. Changed it. Done.
Issue Time Tracking
---
Worklog Id: (was: 852848)
Time Spent: 2h 40m (was: 2.5h)
> Use basename of the datatype when fetching partition metadata using partition
> filters
> -
>
> Key: HIVE-27168
> URL: https://issues.apache.org/jira/browse/HIVE-27168
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> While fetching partition metadata using partition filters, we use the column
> type of the table directly. However, char/varchar types can contain extra
> information such as length of the char/varchar column and hence it skips
> fetching partition metadata due to this extra information.
> Solution: Use the basename of the column type while deciding on whether
> partition pruning can be done on the partitioned column.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[
https://issues.apache.org/jira/browse/HIVE-27168?focusedWorklogId=852845=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852845
]
ASF GitHub Bot logged work on HIVE-27168:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 12:45
Start Date: 24/Mar/23 12:45
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4144:
URL: https://github.com/apache/hive/pull/4144#discussion_r1147526711
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##
@@ -1327,7 +1327,7 @@ public void visit(LeafNode node) throws MetaException {
return;
}
- String colTypeStr = partitionKeys.get(partColIndex).getType();
+ String colTypeStr =
MetaStoreUtils.getBaseNameOfColType(partitionKeys.get(partColIndex).getType());
Review Comment:
i think you can use `ColumnType.getTypeName(colTypeStr)`
Issue Time Tracking
---
Worklog Id: (was: 852845)
Time Spent: 2.5h (was: 2h 20m)
> Use basename of the datatype when fetching partition metadata using partition
> filters
> -
>
> Key: HIVE-27168
> URL: https://issues.apache.org/jira/browse/HIVE-27168
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> While fetching partition metadata using partition filters, we use the column
> type of the table directly. However, char/varchar types can contain extra
> information such as length of the char/varchar column and hence it skips
> fetching partition metadata due to this extra information.
> Solution: Use the basename of the column type while deciding on whether
> partition pruning can be done on the partitioned column.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[
https://issues.apache.org/jira/browse/HIVE-27168?focusedWorklogId=852841=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852841
]
ASF GitHub Bot logged work on HIVE-27168:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 12:39
Start Date: 24/Mar/23 12:39
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #4144:
URL: https://github.com/apache/hive/pull/4144#discussion_r1147527514
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##
@@ -1327,7 +1327,7 @@ public void visit(LeafNode node) throws MetaException {
return;
}
- String colTypeStr = partitionKeys.get(partColIndex).getType();
+ String colTypeStr =
MetaStoreUtils.getBaseNameOfColType(partitionKeys.get(partColIndex).getType());
Review Comment:
Similar implementation is present in TypeInfoUtils#getBaseName(String) which
does this conversion. But this would require adding serde module into
standalone-metastore which seems to me like an overkill for this fix.
Issue Time Tracking
---
Worklog Id: (was: 852841)
Time Spent: 2h 10m (was: 2h)
> Use basename of the datatype when fetching partition metadata using partition
> filters
> -
>
> Key: HIVE-27168
> URL: https://issues.apache.org/jira/browse/HIVE-27168
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> While fetching partition metadata using partition filters, we use the column
> type of the table directly. However, char/varchar types can contain extra
> information such as length of the char/varchar column and hence it skips
> fetching partition metadata due to this extra information.
> Solution: Use the basename of the column type while deciding on whether
> partition pruning can be done on the partitioned column.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[
https://issues.apache.org/jira/browse/HIVE-27168?focusedWorklogId=852842=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852842
]
ASF GitHub Bot logged work on HIVE-27168:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 12:39
Start Date: 24/Mar/23 12:39
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #4144:
URL: https://github.com/apache/hive/pull/4144#discussion_r1147527514
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##
@@ -1327,7 +1327,7 @@ public void visit(LeafNode node) throws MetaException {
return;
}
- String colTypeStr = partitionKeys.get(partColIndex).getType();
+ String colTypeStr =
MetaStoreUtils.getBaseNameOfColType(partitionKeys.get(partColIndex).getType());
Review Comment:
Similar implementation is present in `TypeInfoUtils#getBaseName(String)`
which does this conversion. But this would require adding `serde` module into
`standalone-metastore` which seems to me like an overkill for this fix.
Issue Time Tracking
---
Worklog Id: (was: 852842)
Time Spent: 2h 20m (was: 2h 10m)
> Use basename of the datatype when fetching partition metadata using partition
> filters
> -
>
> Key: HIVE-27168
> URL: https://issues.apache.org/jira/browse/HIVE-27168
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> While fetching partition metadata using partition filters, we use the column
> type of the table directly. However, char/varchar types can contain extra
> information such as length of the char/varchar column and hence it skips
> fetching partition metadata due to this extra information.
> Solution: Use the basename of the column type while deciding on whether
> partition pruning can be done on the partitioned column.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[
https://issues.apache.org/jira/browse/HIVE-27168?focusedWorklogId=852839=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852839
]
ASF GitHub Bot logged work on HIVE-27168:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 12:38
Start Date: 24/Mar/23 12:38
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4144:
URL: https://github.com/apache/hive/pull/4144#discussion_r1147526711
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##
@@ -1327,7 +1327,7 @@ public void visit(LeafNode node) throws MetaException {
return;
}
- String colTypeStr = partitionKeys.get(partColIndex).getType();
+ String colTypeStr =
MetaStoreUtils.getBaseNameOfColType(partitionKeys.get(partColIndex).getType());
Review Comment:
i think you can use `TypeInfoUtils.getTypeInfoFromTypeString(colTypeStr )`
Issue Time Tracking
---
Worklog Id: (was: 852839)
Time Spent: 1h 50m (was: 1h 40m)
> Use basename of the datatype when fetching partition metadata using partition
> filters
> -
>
> Key: HIVE-27168
> URL: https://issues.apache.org/jira/browse/HIVE-27168
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> While fetching partition metadata using partition filters, we use the column
> type of the table directly. However, char/varchar types can contain extra
> information such as length of the char/varchar column and hence it skips
> fetching partition metadata due to this extra information.
> Solution: Use the basename of the column type while deciding on whether
> partition pruning can be done on the partitioned column.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[
https://issues.apache.org/jira/browse/HIVE-27168?focusedWorklogId=852840=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852840
]
ASF GitHub Bot logged work on HIVE-27168:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 12:38
Start Date: 24/Mar/23 12:38
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4144:
URL: https://github.com/apache/hive/pull/4144#discussion_r1147526711
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##
@@ -1327,7 +1327,7 @@ public void visit(LeafNode node) throws MetaException {
return;
}
- String colTypeStr = partitionKeys.get(partColIndex).getType();
+ String colTypeStr =
MetaStoreUtils.getBaseNameOfColType(partitionKeys.get(partColIndex).getType());
Review Comment:
i think you can use `TypeInfo columnType =
TypeInfoUtils.getTypeInfoFromTypeString(colTypeStr )`
Issue Time Tracking
---
Worklog Id: (was: 852840)
Time Spent: 2h (was: 1h 50m)
> Use basename of the datatype when fetching partition metadata using partition
> filters
> -
>
> Key: HIVE-27168
> URL: https://issues.apache.org/jira/browse/HIVE-27168
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> While fetching partition metadata using partition filters, we use the column
> type of the table directly. However, char/varchar types can contain extra
> information such as length of the char/varchar column and hence it skips
> fetching partition metadata due to this extra information.
> Solution: Use the basename of the column type while deciding on whether
> partition pruning can be done on the partitioned column.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[
https://issues.apache.org/jira/browse/HIVE-27168?focusedWorklogId=852834=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852834
]
ASF GitHub Bot logged work on HIVE-27168:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 12:13
Start Date: 24/Mar/23 12:13
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #4144:
URL: https://github.com/apache/hive/pull/4144#discussion_r1147503933
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##
@@ -1251,4 +1251,19 @@ public static String getHttpPath(String httpPath) {
}
return httpPath;
}
+
+ /**
+ * Function to get the base name of the column type.
+ * @param colType column type
+ * @return base name of the column type
+ */
+ public static String getBaseNameOfColType(String colType) {
+// char, varchar types can have parameters such as
+// char/varchar length. We only need the base name.
+int idx = colType.indexOf('(');
Review Comment:
Added trims as well just for safety if we encounter such a situation.
Issue Time Tracking
---
Worklog Id: (was: 852834)
Time Spent: 1h 40m (was: 1.5h)
> Use basename of the datatype when fetching partition metadata using partition
> filters
> -
>
> Key: HIVE-27168
> URL: https://issues.apache.org/jira/browse/HIVE-27168
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> While fetching partition metadata using partition filters, we use the column
> type of the table directly. However, char/varchar types can contain extra
> information such as length of the char/varchar column and hence it skips
> fetching partition metadata due to this extra information.
> Solution: Use the basename of the column type while deciding on whether
> partition pruning can be done on the partitioned column.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[
https://issues.apache.org/jira/browse/HIVE-27168?focusedWorklogId=852829=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852829
]
ASF GitHub Bot logged work on HIVE-27168:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 11:33
Start Date: 24/Mar/23 11:33
Worklog Time Spent: 10m
Work Description: veghlaci05 commented on code in PR #4144:
URL: https://github.com/apache/hive/pull/4144#discussion_r1147456482
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##
@@ -1251,4 +1251,19 @@ public static String getHttpPath(String httpPath) {
}
return httpPath;
}
+
+ /**
+ * Function to get the base name of the column type.
+ * @param colType column type
+ * @return base name of the column type
+ */
+ public static String getBaseNameOfColType(String colType) {
+// char, varchar types can have parameters such as
+// char/varchar length. We only need the base name.
+int idx = colType.indexOf('(');
+if (idx != -1) {
+ return colType.substring(0, idx);
Review Comment:
Is coltype already trimmed here? what if there are whitespaces at the end?
For example `varchar(30) `
If this could be an issue, a test case also should be added in
`TestHiveMetaStore.testPartitionFilter()`
Issue Time Tracking
---
Worklog Id: (was: 852829)
Time Spent: 1.5h (was: 1h 20m)
> Use basename of the datatype when fetching partition metadata using partition
> filters
> -
>
> Key: HIVE-27168
> URL: https://issues.apache.org/jira/browse/HIVE-27168
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> While fetching partition metadata using partition filters, we use the column
> type of the table directly. However, char/varchar types can contain extra
> information such as length of the char/varchar column and hence it skips
> fetching partition metadata due to this extra information.
> Solution: Use the basename of the column type while deciding on whether
> partition pruning can be done on the partitioned column.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27169) New Locked List to prevent configuration change at runtime without throwing error
[ https://issues.apache.org/jira/browse/HIVE-27169?focusedWorklogId=852822=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852822 ] ASF GitHub Bot logged work on HIVE-27169: - Author: ASF GitHub Bot Created on: 24/Mar/23 10:42 Start Date: 24/Mar/23 10:42 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4146: URL: https://github.com/apache/hive/pull/4146#issuecomment-1482595443 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4146) [](https://sonarcloud.io/project/issues?id=apache_hive=4146=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4146=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4146=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4146=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4146=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4146=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4146=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4146=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4146=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4146=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4146=false=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=4146=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4146=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4146=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 852822) Time Spent: 0.5h (was: 20m) > New Locked List to prevent configuration change at runtime without throwing > error > - > > Key: HIVE-27169 > URL: https://issues.apache.org/jira/browse/HIVE-27169 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0-alpha-2 >Reporter: Raghav Aggarwal >Assignee: Raghav Aggarwal >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > _*AIM*_ > Create a new locked list called _*hive.conf.locked.list*_ which contains > comma separated configuration that can't be changed during runtime. If > someone try to change them at runtime then it will give WARN log on beeline > itself and will not change that config. > > _*How is it different from Restricted List?*_ > When running hql file or at runtime, if a configuration present in restricted > list get updated then it will throw error and won't proceed with further > execution of hql file. > With locked list, the configuration that is getting updated will throw > _*WARN*_ log on beeline and will continue to execute the hql file. > > _*Why is it required?*_ > In organisations, admin want to enforce some configs which user shouldn't be > able to change at runtime and it shouldn't affect user's existing hql > scripts. Therefore, this locked list will be useful as it
[jira] [Work logged] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[
https://issues.apache.org/jira/browse/HIVE-27168?focusedWorklogId=852793=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852793
]
ASF GitHub Bot logged work on HIVE-27168:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 09:07
Start Date: 24/Mar/23 09:07
Worklog Time Spent: 10m
Work Description: InvisibleProgrammer commented on code in PR #4144:
URL: https://github.com/apache/hive/pull/4144#discussion_r1147291829
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##
@@ -1251,4 +1251,19 @@ public static String getHttpPath(String httpPath) {
}
return httpPath;
}
+
+ /**
+ * Function to get the base name of the column type.
+ * @param colType column type
+ * @return base name of the column type
+ */
+ public static String getBaseNameOfColType(String colType) {
+// char, varchar types can have parameters such as
+// char/varchar length. We only need the base name.
+int idx = colType.indexOf('(');
Review Comment:
Got it, thx.
Issue Time Tracking
---
Worklog Id: (was: 852793)
Time Spent: 1h 20m (was: 1h 10m)
> Use basename of the datatype when fetching partition metadata using partition
> filters
> -
>
> Key: HIVE-27168
> URL: https://issues.apache.org/jira/browse/HIVE-27168
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> While fetching partition metadata using partition filters, we use the column
> type of the table directly. However, char/varchar types can contain extra
> information such as length of the char/varchar column and hence it skips
> fetching partition metadata due to this extra information.
> Solution: Use the basename of the column type while deciding on whether
> partition pruning can be done on the partitioned column.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[
https://issues.apache.org/jira/browse/HIVE-27168?focusedWorklogId=852790=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852790
]
ASF GitHub Bot logged work on HIVE-27168:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 09:02
Start Date: 24/Mar/23 09:02
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #4144:
URL: https://github.com/apache/hive/pull/4144#discussion_r1147287357
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##
@@ -1251,4 +1251,19 @@ public static String getHttpPath(String httpPath) {
}
return httpPath;
}
+
+ /**
+ * Function to get the base name of the column type.
+ * @param colType column type
+ * @return base name of the column type
+ */
+ public static String getBaseNameOfColType(String colType) {
+// char, varchar types can have parameters such as
+// char/varchar length. We only need the base name.
+int idx = colType.indexOf('(');
Review Comment:
The `colType` that you see here is retrieved from metastore DB. I think its
an optimisation to not maintain 2 column in metastore DB - one for type and one
for length / other information (btw this information is only for char, varchar
or other types).
Also similar implementation is present in
`TypeInfoUtils#getBaseName(String)` which does this conversion. But this would
require adding `serde` module into `standalone-metastore` which seems to me
like an overkill for this fix.
Issue Time Tracking
---
Worklog Id: (was: 852790)
Time Spent: 1h 10m (was: 1h)
> Use basename of the datatype when fetching partition metadata using partition
> filters
> -
>
> Key: HIVE-27168
> URL: https://issues.apache.org/jira/browse/HIVE-27168
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> While fetching partition metadata using partition filters, we use the column
> type of the table directly. However, char/varchar types can contain extra
> information such as length of the char/varchar column and hence it skips
> fetching partition metadata due to this extra information.
> Solution: Use the basename of the column type while deciding on whether
> partition pruning can be done on the partitioned column.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[
https://issues.apache.org/jira/browse/HIVE-27168?focusedWorklogId=852789=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852789
]
ASF GitHub Bot logged work on HIVE-27168:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 09:02
Start Date: 24/Mar/23 09:02
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #4144:
URL: https://github.com/apache/hive/pull/4144#discussion_r1147287357
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##
@@ -1251,4 +1251,19 @@ public static String getHttpPath(String httpPath) {
}
return httpPath;
}
+
+ /**
+ * Function to get the base name of the column type.
+ * @param colType column type
+ * @return base name of the column type
+ */
+ public static String getBaseNameOfColType(String colType) {
+// char, varchar types can have parameters such as
+// char/varchar length. We only need the base name.
+int idx = colType.indexOf('(');
Review Comment:
The `colType` that you see here is retrieved from metastore DB. I think its
an optimisation to not maintain 2 column in metastore DB - one for type and one
for length / other information (btw this information is only for char, varchar
or other types).
Also similar implementation is present in TypeInfoUtils#getBaseName(String)
which does this conversion. But this would require adding `serde` module into
`standalone-metastore` which seems to me like an overkill for this fix.
Issue Time Tracking
---
Worklog Id: (was: 852789)
Time Spent: 1h (was: 50m)
> Use basename of the datatype when fetching partition metadata using partition
> filters
> -
>
> Key: HIVE-27168
> URL: https://issues.apache.org/jira/browse/HIVE-27168
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> While fetching partition metadata using partition filters, we use the column
> type of the table directly. However, char/varchar types can contain extra
> information such as length of the char/varchar column and hence it skips
> fetching partition metadata due to this extra information.
> Solution: Use the basename of the column type while deciding on whether
> partition pruning can be done on the partitioned column.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-20680) Bootstrap is missing partitions in replicated DB when retry after kill query.
[ https://issues.apache.org/jira/browse/HIVE-20680?focusedWorklogId=852781=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852781 ] ASF GitHub Bot logged work on HIVE-20680: - Author: ASF GitHub Bot Created on: 24/Mar/23 08:30 Start Date: 24/Mar/23 08:30 Worklog Time Spent: 10m Work Description: vihangk1 opened a new pull request, #4148: URL: https://github.com/apache/hive/pull/4148 …ry after kill query (Mahesh Kumar Behera, reviewed by Sankar Hariappan) ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Issue Time Tracking --- Worklog Id: (was: 852781) Time Spent: 0.5h (was: 20m) > Bootstrap is missing partitions in replicated DB when retry after kill query. > - > > Key: HIVE-20680 > URL: https://issues.apache.org/jira/browse/HIVE-20680 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Attachments: HIVE-20680.01.patch, HIVE-20680.02.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > The issue is > 1. When bootstrap was going on, kill query was called to kill the repl load > command. > 2. During restart, one table with no partition set the scope to table as the > ckpt property was not yet set for that table. > 3. Due to this, all partitioned table after this didn't get their tasks > related to partition appended in the root task if the ckpt property is > already updated for those tables. > The fix is to reset the table scope to false if for that table there are no > tasks added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27169) New Locked List to prevent configuration change at runtime without throwing error
[
https://issues.apache.org/jira/browse/HIVE-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raghav Aggarwal updated HIVE-27169:
---
Description:
_*AIM*_
Create a new locked list called _*hive.conf.locked.list*_ which contains comma
separated configuration that can't be changed during runtime. If someone try to
change them at runtime then it will give WARN log on beeline itself and will
not change that config.
_*How is it different from Restricted List?*_
When running hql file or at runtime, if a configuration present in restricted
list get updated then it will throw error and won't proceed with further
execution of hql file.
With locked list, the configuration that is getting updated will throw _*WARN*_
log on beeline and will continue to execute the hql file.
_*Why is it required?*_
In organisations, admin want to enforce some configs which user shouldn't be
able to change at runtime and it shouldn't affect user's existing hql scripts.
Therefore, this locked list will be useful as it will not allow user to change
the value of particular configs and it will also not stop the execution of hql
scripts.
{_}*NOTE*{_}: Only at cluster level _*hive.conf.locked.list*_ can be set and
after that the hive service needs to be restarted.
This will be very helpful when organisations are migrating from Hive 1.x,
Hive2.x to higher version and admin want to enforce some configuration which
should remain constant.
was:
_*AIM*_
Create a new locked list called _*hive.conf.locked.list*_ which contains comma
separated configuration that won't be changed during runtime. If someone try to
change them at runtime then it will give WARN log on beeline itself.
_*How is it different from Restricted List?*_
When running hql file or at runtime, if a configuration present in restricted
list get updated then it will throw error and won't proceed with further
execution of hql file.
With locked list, the configuration that is getting updated will throw _*WARN*_
log on beeline and will continue to execute the hql file.
_*Why is it required?*_
In organisations, admin want to enforce some configs which user shouldn't be
able to change at runtime and it shouldn't affect user's existing hql scripts.
Therefore, this locked list will be useful as it will not allow user to change
the value of particular configs and it will also not stop the execution of hql
scripts.
{_}*NOTE*{_}: Only at cluster level _*hive.conf.locked.list*_ can be set and
after that the hive service needs to be restarted.
> New Locked List to prevent configuration change at runtime without throwing
> error
> -
>
> Key: HIVE-27169
> URL: https://issues.apache.org/jira/browse/HIVE-27169
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0-alpha-2
>Reporter: Raghav Aggarwal
>Assignee: Raghav Aggarwal
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> _*AIM*_
> Create a new locked list called _*hive.conf.locked.list*_ which contains
> comma separated configuration that can't be changed during runtime. If
> someone try to change them at runtime then it will give WARN log on beeline
> itself and will not change that config.
>
> _*How is it different from Restricted List?*_
> When running hql file or at runtime, if a configuration present in restricted
> list get updated then it will throw error and won't proceed with further
> execution of hql file.
> With locked list, the configuration that is getting updated will throw
> _*WARN*_ log on beeline and will continue to execute the hql file.
>
> _*Why is it required?*_
> In organisations, admin want to enforce some configs which user shouldn't be
> able to change at runtime and it shouldn't affect user's existing hql
> scripts. Therefore, this locked list will be useful as it will not allow user
> to change the value of particular configs and it will also not stop the
> execution of hql scripts.
>
> {_}*NOTE*{_}: Only at cluster level _*hive.conf.locked.list*_ can be set and
> after that the hive service needs to be restarted.
> This will be very helpful when organisations are migrating from Hive 1.x,
> Hive2.x to higher version and admin want to enforce some configuration which
> should remain constant.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27158) Store hive columns stats in puffin files for iceberg tables
[ https://issues.apache.org/jira/browse/HIVE-27158?focusedWorklogId=852779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852779 ] ASF GitHub Bot logged work on HIVE-27158: - Author: ASF GitHub Bot Created on: 24/Mar/23 08:23 Start Date: 24/Mar/23 08:23 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4131: URL: https://github.com/apache/hive/pull/4131#issuecomment-1482423092 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4131) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4131=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4131=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4131=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=CODE_SMELL) [6 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=4131=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4131=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4131=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 852779) Time Spent: 1h (was: 50m) > Store hive columns stats in puffin files for iceberg tables > --- > > Key: HIVE-27158 > URL: https://issues.apache.org/jira/browse/HIVE-27158 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[
https://issues.apache.org/jira/browse/HIVE-27168?focusedWorklogId=852776=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852776
]
ASF GitHub Bot logged work on HIVE-27168:
-
Author: ASF GitHub Bot
Created on: 24/Mar/23 08:13
Start Date: 24/Mar/23 08:13
Worklog Time Spent: 10m
Work Description: InvisibleProgrammer commented on code in PR #4144:
URL: https://github.com/apache/hive/pull/4144#discussion_r1147241022
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##
@@ -1251,4 +1251,19 @@ public static String getHttpPath(String httpPath) {
}
return httpPath;
}
+
+ /**
+ * Function to get the base name of the column type.
+ * @param colType column type
+ * @return base name of the column type
+ */
+ public static String getBaseNameOfColType(String colType) {
+// char, varchar types can have parameters such as
+// char/varchar length. We only need the base name.
+int idx = colType.indexOf('(');
Review Comment:
Can it be stored with extra whitespace characters, like `varchar (50)`?
To be honest, I'm a pretty surprised that we don't have existing code to
split the type name and the length.
Issue Time Tracking
---
Worklog Id: (was: 852776)
Time Spent: 50m (was: 40m)
> Use basename of the datatype when fetching partition metadata using partition
> filters
> -
>
> Key: HIVE-27168
> URL: https://issues.apache.org/jira/browse/HIVE-27168
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> While fetching partition metadata using partition filters, we use the column
> type of the table directly. However, char/varchar types can contain extra
> information such as length of the char/varchar column and hence it skips
> fetching partition metadata due to this extra information.
> Solution: Use the basename of the column type while deciding on whether
> partition pruning can be done on the partitioned column.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-21753) Update HiveMetastore authorization to enable use of HiveAuthorizer implementation
[
https://issues.apache.org/jira/browse/HIVE-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17704504#comment-17704504
]
YUBI LEE commented on HIVE-21753:
-
[~rmani]
Hello, first of all, thanks for this contribution.
I just want to ask a question.
Does it work with kerberos enabled?
I keep encountering Kerberos error when I access to HiveMetastore through
spark-sql.
This is a log from HiveMetastore.
{code}
Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Could not
connect to meta store using any of the URIs provided. Most recent failure:
org.apache.thrift.transport.TTransportException: GSS initiate failed
at
org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:51)
at
org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:48)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at
org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport.open(TUGIAssumingTransport.java:48)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:516)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:224)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:94)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:84)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:95)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4306)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4374)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4354)
at
org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4610)
at
org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:291)
at
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:274)
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:442)
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:382)
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:362)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:408)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:404)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveMetastoreClientFactoryImpl.getHiveMetastoreClient(HiveMetastoreClientFactoryImpl.java:36)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizationValidator.checkPrivileges(SQLStdHiveAuthorizationValidator.java:79)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerImpl.checkPrivileges(HiveAuthorizerImpl.java:86)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.checkPrivileges(HiveMetaStoreAuthorizer.java:245)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.onEvent(HiveMetaStoreAuthorizer.java:109)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(HiveMetaStore.java:3222)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:5030)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:4997)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
[jira] [Updated] (HIVE-27169) New Locked List to prevent configuration change at runtime without throwing error
[
https://issues.apache.org/jira/browse/HIVE-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raghav Aggarwal updated HIVE-27169:
---
Description:
_*AIM*_
Create a new locked list called _*hive.conf.locked.list*_ which contains comma
separated configuration that won't be changed during runtime. If someone try to
change them at runtime then it will give WARN log on beeline itself.
_*How is it different from Restricted List?*_
When running hql file or at runtime, if a configuration present in restricted
list get updated then it will throw error and won't proceed with further
execution of hql file.
With locked list, the configuration that is getting updated will throw _*WARN*_
log on beeline and will continue to execute the hql file.
_*Why is it required?*_
In organisations, admin want to enforce some configs which user shouldn't be
able to change at runtime and it shouldn't affect user's existing hql scripts.
Therefore, this locked list will be useful as it will not allow user to change
the value of particular configs and it will also not stop the execution of hql
scripts.
{_}*NOTE*{_}: Only at cluster level _*hive.conf.locked.list*_ can be set and
after that the hive service needs to be restarted.
was:
_*AIM*_
Create a new locked list called\{{ hive.conf.locked.list}} which contains comma
separated configuration that won't be changed during runtime. If someone try to
change them at runtime then it will give WARN log on beeline itself.
_*How is it different from Restricted List?*_
When running hql file or at runtime, if a configuration present in restricted
list get updated then it will throw error and won't proceed with further
execution of hql file.
With locked list, the configuration that is getting updated will throw WARN log
on beeline and will continue to execute the hql file.
_*Why is it required?*_
In organisations, admin want to enforce some configs which user shouldn't be
able to change at runtime and it shouldn't affect user's existing hql scripts.
Therefore, this locked list will be useful as it will not allow user to change
the value of particular configs and it will also not stop the execution of hql
scripts.
{_}*NOTE*{_}: Only at cluster level _*hive.conf.locked.list*_ can be set and
after that the hive service needs to be restarted.
> New Locked List to prevent configuration change at runtime without throwing
> error
> -
>
> Key: HIVE-27169
> URL: https://issues.apache.org/jira/browse/HIVE-27169
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0-alpha-2
>Reporter: Raghav Aggarwal
>Assignee: Raghav Aggarwal
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> _*AIM*_
> Create a new locked list called _*hive.conf.locked.list*_ which contains
> comma separated configuration that won't be changed during runtime. If
> someone try to change them at runtime then it will give WARN log on beeline
> itself.
>
> _*How is it different from Restricted List?*_
> When running hql file or at runtime, if a configuration present in restricted
> list get updated then it will throw error and won't proceed with further
> execution of hql file.
> With locked list, the configuration that is getting updated will throw
> _*WARN*_ log on beeline and will continue to execute the hql file.
>
> _*Why is it required?*_
> In organisations, admin want to enforce some configs which user shouldn't be
> able to change at runtime and it shouldn't affect user's existing hql
> scripts. Therefore, this locked list will be useful as it will not allow user
> to change the value of particular configs and it will also not stop the
> execution of hql scripts.
>
> {_}*NOTE*{_}: Only at cluster level _*hive.conf.locked.list*_ can be set and
> after that the hive service needs to be restarted.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27169) New Locked List to prevent configuration change at runtime without throwing error
[
https://issues.apache.org/jira/browse/HIVE-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raghav Aggarwal updated HIVE-27169:
---
Description:
_*AIM*_
Create a new locked list called\{{ hive.conf.locked.list}} which contains comma
separated configuration that won't be changed during runtime. If someone try to
change them at runtime then it will give WARN log on beeline itself.
_*How is it different from Restricted List?*_
When running hql file or at runtime, if a configuration present in restricted
list get updated then it will throw error and won't proceed with further
execution of hql file.
With locked list, the configuration that is getting updated will throw WARN log
on beeline and will continue to execute the hql file.
_*Why is it required?*_
In organisations, admin want to enforce some configs which user shouldn't be
able to change at runtime and it shouldn't affect user's existing hql scripts.
Therefore, this locked list will be useful as it will not allow user to change
the value of particular configs and it will also not stop the execution of hql
scripts.
{_}*NOTE*{_}: Only at cluster level _*hive.conf.locked.list*_ can be set and
after that the hive service needs to be restarted.
was:
_*AIM*_
Create a new locked list called{{ hive.conf.locked.list}} which contains comma
separated configuration that won't be changed during runtime. If someone try to
change them at runtime then it will give WARN log on beeline itself.
_*How is it different from Restricted List?*_
When running hql file or at runtime, if a configuration present in restricted
list get updated then it will throw error and won't proceed with further
execution of hql file.
With locked list, the configuration that is getting updated will throw WARN log
on beeline and will continue to execute the hql file.
_*Why is it required?*_
In organisations, admin want to enforce some configs which user shouldn't be
able to change at runtime and it shouldn't affect user's existing hql scripts.
Therefore, this locked list will be useful as it will not allow user to change
the value of particular configs and it will also not stop the execution of hql
scripts.
{_}*NOTE*{_}: Only at cluster level {{hive.conf.locked.list }}can be set and
after that the hive service needs to be restarted.
> New Locked List to prevent configuration change at runtime without throwing
> error
> -
>
> Key: HIVE-27169
> URL: https://issues.apache.org/jira/browse/HIVE-27169
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0-alpha-2
>Reporter: Raghav Aggarwal
>Assignee: Raghav Aggarwal
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> _*AIM*_
> Create a new locked list called\{{ hive.conf.locked.list}} which contains
> comma separated configuration that won't be changed during runtime. If
> someone try to change them at runtime then it will give WARN log on beeline
> itself.
>
> _*How is it different from Restricted List?*_
> When running hql file or at runtime, if a configuration present in restricted
> list get updated then it will throw error and won't proceed with further
> execution of hql file.
> With locked list, the configuration that is getting updated will throw WARN
> log on beeline and will continue to execute the hql file.
>
> _*Why is it required?*_
> In organisations, admin want to enforce some configs which user shouldn't be
> able to change at runtime and it shouldn't affect user's existing hql
> scripts. Therefore, this locked list will be useful as it will not allow user
> to change the value of particular configs and it will also not stop the
> execution of hql scripts.
>
> {_}*NOTE*{_}: Only at cluster level _*hive.conf.locked.list*_ can be set and
> after that the hive service needs to be restarted.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27168) Use basename of the datatype when fetching partition metadata using partition filters
[ https://issues.apache.org/jira/browse/HIVE-27168?focusedWorklogId=852768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852768 ] ASF GitHub Bot logged work on HIVE-27168: - Author: ASF GitHub Bot Created on: 24/Mar/23 07:42 Start Date: 24/Mar/23 07:42 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4144: URL: https://github.com/apache/hive/pull/4144#issuecomment-1482381207 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=4144) [](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4144=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4144=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4144=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=4144=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=4144=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=4144=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 852768) Time Spent: 40m (was: 0.5h) > Use basename of the datatype when fetching partition metadata using partition > filters > - > > Key: HIVE-27168 > URL: https://issues.apache.org/jira/browse/HIVE-27168 > Project: Hive > Issue Type: Bug >Reporter: Sourabh Badhya >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > While fetching partition metadata using partition filters, we use the column > type of the table directly. However, char/varchar types can contain extra > information such as length of the char/varchar column and hence it skips > fetching partition metadata due to this extra information. > Solution: Use the basename of the column type while deciding on whether > partition pruning can be done on the partitioned column. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26400) Provide docker images for Hive
[ https://issues.apache.org/jira/browse/HIVE-26400?focusedWorklogId=852758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-852758 ] ASF GitHub Bot logged work on HIVE-26400: - Author: ASF GitHub Bot Created on: 24/Mar/23 06:20 Start Date: 24/Mar/23 06:20 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3448: URL: https://github.com/apache/hive/pull/3448#issuecomment-1482310775 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3448) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3448=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3448=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3448=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3448=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3448=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 852758) Time Spent: 9h 50m (was: 9h 40m) > Provide docker images for Hive > -- > > Key: HIVE-26400 > URL: https://issues.apache.org/jira/browse/HIVE-26400 > Project: Hive > Issue Type: Sub-task > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Blocker > Labels: hive-4.0.0-must, pull-request-available > Time Spent: 9h 50m > Remaining Estimate: 0h > > Make Apache Hive be able to run inside docker container in pseudo-distributed > mode, with MySQL/Derby as its back database, provide the following: > * Quick-start/Debugging/Prepare a test env for Hive; > * Tools to build target image with specified version of Hive and its > dependencies; > * Images can be used as the basis for the Kubernetes operator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
