[jira] [Resolved] (HIVE-23185) Historic queries lost after HS2 restart
[ https://issues.apache.org/jira/browse/HIVE-23185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng resolved HIVE-23185. Resolution: Won't Fix > Historic queries lost after HS2 restart > --- > > Key: HIVE-23185 > URL: https://issues.apache.org/jira/browse/HIVE-23185 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > > QueryInfoCache caches historic queries in memory, when HS2 restart due to OOM > or upgrade, the queries are no longer seen at webui. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-24511) Potential classloader leak in SerDeStorageSchemaReader and add JsonSerde to managed serde
[ https://issues.apache.org/jira/browse/HIVE-24511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng resolved HIVE-24511. Resolution: Incomplete > Potential classloader leak in SerDeStorageSchemaReader and add JsonSerde to > managed serde > - > > Key: HIVE-24511 > URL: https://issues.apache.org/jira/browse/HIVE-24511 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > 1, Close the created classloader to release resources. > 2, More detail error messages on MetaException when throwing. > 3, Skip JsonSerDe/RegexSerDe creation when get columns/schemas of such > tables. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25688) Non blocking DROP PARTITION implementation
[ https://issues.apache.org/jira/browse/HIVE-25688?focusedWorklogId=703817&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-703817 ] ASF GitHub Bot logged work on HIVE-25688: - Author: ASF GitHub Bot Created on: 05/Jan/22 08:54 Start Date: 05/Jan/22 08:54 Worklog Time Spent: 10m Work Description: deniskuzZ merged pull request #2780: URL: https://github.com/apache/hive/pull/2780 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 703817) Time Spent: 4h 20m (was: 4h 10m) > Non blocking DROP PARTITION implementation > -- > > Key: HIVE-25688 > URL: https://issues.apache.org/jira/browse/HIVE-25688 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 4h 20m > Remaining Estimate: 0h > > This changes the drop partition operation to asynchronous. The data files of > transactional tables will not be deleted, but a new truncated basefile will > be written, which is going to be later cleaned up by the Compactor/Cleaner. > This along with a few other changes will enable us to not use read locks, > which provides perf boost to the transactional tables. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25688) Non blocking DROP PARTITION implementation
[ https://issues.apache.org/jira/browse/HIVE-25688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko resolved HIVE-25688. --- Assignee: Denys Kuzmenko Resolution: Fixed > Non blocking DROP PARTITION implementation > -- > > Key: HIVE-25688 > URL: https://issues.apache.org/jira/browse/HIVE-25688 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 4h 20m > Remaining Estimate: 0h > > This changes the drop partition operation to asynchronous. The data files of > transactional tables will not be deleted, but a new truncated basefile will > be written, which is going to be later cleaned up by the Compactor/Cleaner. > This along with a few other changes will enable us to not use read locks, > which provides perf boost to the transactional tables. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25688) Non blocking DROP PARTITION implementation
[ https://issues.apache.org/jira/browse/HIVE-25688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469132#comment-17469132 ] Denys Kuzmenko commented on HIVE-25688: --- Merged to master. [~pvary] thank you for the review! > Non blocking DROP PARTITION implementation > -- > > Key: HIVE-25688 > URL: https://issues.apache.org/jira/browse/HIVE-25688 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 4h 20m > Remaining Estimate: 0h > > This changes the drop partition operation to asynchronous. The data files of > transactional tables will not be deleted, but a new truncated basefile will > be written, which is going to be later cleaned up by the Compactor/Cleaner. > This along with a few other changes will enable us to not use read locks, > which provides perf boost to the transactional tables. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-24805) Compactor: Initiator shouldn't fetch table details again and again for partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-24805?focusedWorklogId=703830&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-703830
]
ASF GitHub Bot logged work on HIVE-24805:
-
Author: ASF GitHub Bot
Created on: 05/Jan/22 09:24
Start Date: 05/Jan/22 09:24
Worklog Time Spent: 10m
Work Description: asinkovits commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r778663909
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##
@@ -201,7 +201,7 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB,
boolean metricsEnabled
Partition p = null;
if (ci.partName != null) {
-p = resolvePartition(ci);
+p = cacheAndResolvePartition(ci);
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 703830)
Time Spent: 1h 20m (was: 1h 10m)
> Compactor: Initiator shouldn't fetch table details again and again for
> partitioned tables
> -
>
> Key: HIVE-24805
> URL: https://issues.apache.org/jira/browse/HIVE-24805
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Rajesh Balamohan
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Initiator shouldn't be fetch table details for all its partitions. When there
> are large number of databases/tables, it takes lot of time for Initiator to
> complete its initial iteration and load on DB also goes higher.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L129
> https://github.com/apache/hive/blob/64bb52316f19426ebea0087ee15e282cbde1d852/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L456
> For all the following partitions, table details would be the same. However,
> it ends up fetching table details from HMS again and again.
> {noformat}
> 2021-02-22 08:13:16,106 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451899
> 2021-02-22 08:13:16,124 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451830
> 2021-02-22 08:13:16,140 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452586
> 2021-02-22 08:13:16,149 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452698
> 2021-02-22 08:13:16,158 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452063
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24805) Compactor: Initiator shouldn't fetch table details again and again for partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-24805?focusedWorklogId=703829&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-703829
]
ASF GitHub Bot logged work on HIVE-24805:
-
Author: ASF GitHub Bot
Created on: 05/Jan/22 09:24
Start Date: 05/Jan/22 09:24
Worklog Time Spent: 10m
Work Description: asinkovits commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r778663702
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##
@@ -184,7 +184,7 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB,
boolean metricsEnabled
if (metricsEnabled) {
perfLogger.perfLogBegin(CLASS_NAME, cleanerMetric);
}
- Table t = resolveTable(ci);
+ Table t = cacheAndResolveTable(ci);
Review comment:
Sounds reasonable, fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 703829)
Time Spent: 1h 10m (was: 1h)
> Compactor: Initiator shouldn't fetch table details again and again for
> partitioned tables
> -
>
> Key: HIVE-24805
> URL: https://issues.apache.org/jira/browse/HIVE-24805
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Rajesh Balamohan
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Initiator shouldn't be fetch table details for all its partitions. When there
> are large number of databases/tables, it takes lot of time for Initiator to
> complete its initial iteration and load on DB also goes higher.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L129
> https://github.com/apache/hive/blob/64bb52316f19426ebea0087ee15e282cbde1d852/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L456
> For all the following partitions, table details would be the same. However,
> it ends up fetching table details from HMS again and again.
> {noformat}
> 2021-02-22 08:13:16,106 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451899
> 2021-02-22 08:13:16,124 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451830
> 2021-02-22 08:13:16,140 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452586
> 2021-02-22 08:13:16,149 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452698
> 2021-02-22 08:13:16,158 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452063
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24805) Compactor: Initiator shouldn't fetch table details again and again for partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-24805?focusedWorklogId=703833&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-703833
]
ASF GitHub Bot logged work on HIVE-24805:
-
Author: ASF GitHub Bot
Created on: 05/Jan/22 09:26
Start Date: 05/Jan/22 09:26
Worklog Time Spent: 10m
Work Description: asinkovits commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r778664718
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CacheAwareCompactor.java
##
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+
+import java.util.Objects;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+public interface CacheAwareCompactor {
+
+ static void trySetCache(Object obj, CompactorMetadataCache cache) {
+if (CacheAwareCompactor.class.isAssignableFrom(obj.getClass())) {
+ ((CacheAwareCompactor) obj).setCache(cache);
+}
+ }
+
+ void setCache(CompactorMetadataCache cache);
+
+ class CompactorMetadataCache {
+
+private final Cache tableCache;
+private final Cache partitionCache;
+
+@VisibleForTesting
+public CompactorMetadataCache(long timeout, TimeUnit unit) {
+ this.tableCache = CacheBuilder.newBuilder().expireAfterAccess(timeout,
unit).softValues().build();
+ this.partitionCache =
CacheBuilder.newBuilder().expireAfterAccess(timeout, unit).softValues().build();
+}
+
+public static CompactorMetadataCache createIfEnabled(Configuration conf) {
+ long timeout = MetastoreConf.getTimeVar(conf,
+MetastoreConf.ConfVars.COMPACTOR_METADATA_CACHE_TIMEOUT,
TimeUnit.SECONDS);
+ if (timeout == 0) {
+return null;
+ }
+ return new CompactorMetadataCache(timeout, TimeUnit.SECONDS);
+}
+
+public Table resolveTable(CompactionInfo ci, Callable loader) {
+ try {
+TableCacheKey key = new TableCacheKey(ci);
Review comment:
Agree, fixed.
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CacheAwareCompactor.java
##
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+
+import java.util.Objects;
+import java.util.concurrent.Callable;
+import java.util.concur
[jira] [Work logged] (HIVE-24805) Compactor: Initiator shouldn't fetch table details again and again for partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-24805?focusedWorklogId=703835&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-703835
]
ASF GitHub Bot logged work on HIVE-24805:
-
Author: ASF GitHub Bot
Created on: 05/Jan/22 09:29
Start Date: 05/Jan/22 09:29
Worklog Time Spent: 10m
Work Description: asinkovits commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r778667658
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
##
@@ -101,6 +103,25 @@ public void init(AtomicBoolean stop) throws Exception {
}
}
+ @Override
+ public void setCache(CompactorMetadataCache metadataCache) {
+this.metadataCache = metadataCache;
+ }
+
+ protected Table cacheAndResolveTable(CompactionInfo ci) throws MetaException
{
+if (metadataCache != null) {
Review comment:
Well I was thinking this previously and because there was no default
value for the cache, I decided to allow it as null if absent. But you are right
in a sense, that using optional explicitly says that the cache can be null.
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 703835)
Time Spent: 1h 40m (was: 1.5h)
> Compactor: Initiator shouldn't fetch table details again and again for
> partitioned tables
> -
>
> Key: HIVE-24805
> URL: https://issues.apache.org/jira/browse/HIVE-24805
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Rajesh Balamohan
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Initiator shouldn't be fetch table details for all its partitions. When there
> are large number of databases/tables, it takes lot of time for Initiator to
> complete its initial iteration and load on DB also goes higher.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L129
> https://github.com/apache/hive/blob/64bb52316f19426ebea0087ee15e282cbde1d852/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L456
> For all the following partitions, table details would be the same. However,
> it ends up fetching table details from HMS again and again.
> {noformat}
> 2021-02-22 08:13:16,106 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451899
> 2021-02-22 08:13:16,124 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451830
> 2021-02-22 08:13:16,140 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452586
> 2021-02-22 08:13:16,149 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452698
> 2021-02-22 08:13:16,158 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452063
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25783) Enforce ASF headers on Metastore
[ https://issues.apache.org/jira/browse/HIVE-25783?focusedWorklogId=703851&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-703851 ] ASF GitHub Bot logged work on HIVE-25783: - Author: ASF GitHub Bot Created on: 05/Jan/22 10:19 Start Date: 05/Jan/22 10:19 Worklog Time Spent: 10m Work Description: pvary commented on pull request #2852: URL: https://github.com/apache/hive/pull/2852#issuecomment-100849 All of the above points are good. Can we change the jira title/commit message to something which better describes the changes? Thanks, Peter -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 703851) Time Spent: 1h (was: 50m) > Enforce ASF headers on Metastore > > > Key: HIVE-25783 > URL: https://issues.apache.org/jira/browse/HIVE-25783 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > The Jira tries to investigate if we can provide rat check to the CI, make > sure that the newly added source files contain the ASF license information. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25844) Exception deserialization error-s may cause beeline to terminate immediately
[
https://issues.apache.org/jira/browse/HIVE-25844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469193#comment-17469193
]
Stamatis Zampetakis commented on HIVE-25844:
Instead of creating a specific fix for Hive 3.X wouldn't be better to backport
HIVE-24772? As the latter indicates in general it is a bad practice to pass
stack traces to the client. Instead of trying to fix this maybe it is simpler
to remove this "feature". What do you think [~kgyrtkirk]?
> Exception deserialization error-s may cause beeline to terminate immediately
>
>
> Key: HIVE-25844
> URL: https://issues.apache.org/jira/browse/HIVE-25844
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.1.2
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> the exception on the server side happens:
> * fetch task conversion is on
> * there is an exception during reading the table the error bubbles up
> * => transmits a message to beeline that error class name is:
> "org.apache.phoenix.schema.ColumnNotFoundException" + the message
> * it tries to reconstruct the exception around HiveSqlException
> * but during the constructor call
> org.apache.phoenix.exception.SQLExceptionCode is needed which fails to load
> org/apache/hadoop/hbase/shaded/com/google/protobuf/Service
> * a
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/shaded/com/google/protobuf/Service is thrown - which
> is not handled in that method - so it becomes a real error ; and shuts down
> the client
> {code:java}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/shaded/com/google/protobuf/Service
> [...]
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.hive.service.cli.HiveSQLException.newInstance(HiveSQLException.java:245)
> at
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:211)
> [...]
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.shaded.com.google.protobuf.Service
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25783) Improve rat checking in metastore
[ https://issues.apache.org/jira/browse/HIVE-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-25783: --- Summary: Improve rat checking in metastore (was: Enforce ASF headers on Metastore) > Improve rat checking in metastore > -- > > Key: HIVE-25783 > URL: https://issues.apache.org/jira/browse/HIVE-25783 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > The Jira tries to investigate if we can provide rat check to the CI, make > sure that the newly added source files contain the ASF license information. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25667) Unify code managing JDBC databases in tests
[ https://issues.apache.org/jira/browse/HIVE-25667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Bathori updated HIVE-25667: Status: Patch Available (was: Open) > Unify code managing JDBC databases in tests > --- > > Key: HIVE-25667 > URL: https://issues.apache.org/jira/browse/HIVE-25667 > Project: Hive > Issue Type: Task > Components: Testing Infrastructure >Affects Versions: 4.0.0 >Reporter: Stamatis Zampetakis >Assignee: Mark Bathori >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently there are two class hierarchies managing JDBC databases in tests, > [DatabaseRule|https://github.com/apache/hive/blob/d35de014dd49fdcfe0aacb68e6c587beff6d1dea/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/DatabaseRule.java] > and > [AbstractExternalDB|https://github.com/apache/hive/blob/d35de014dd49fdcfe0aacb68e6c587beff6d1dea/itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java]. > There are many similarities between these hierarchies and certain parts are > duplicated. > The goal of this JIRA is to refactor the aforementioned hierarchies to reduce > code duplication and improve extensibility. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-23971) Cleanup unreleased method signatures in IMetastoreClient
[
https://issues.apache.org/jira/browse/HIVE-23971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ping Xu reassigned HIVE-23971:
--
Assignee: Ping Xu (was: Vihang Karajgaonkar)
> Cleanup unreleased method signatures in IMetastoreClient
>
>
> Key: HIVE-23971
> URL: https://issues.apache.org/jira/browse/HIVE-23971
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Ping Xu
>Priority: Blocker
>
> There are many methods in IMetastoreClient which are simply wrappers around
> another method. The code has become very intertwined and needs some cleanup.
> For instance, I see the following variations of {{getPartitionsByNames}} in
> {{IMetastoreClient}}
> {noformat}
> List getPartitionsByNames(String db_name, String tbl_name,
> List part_names, boolean getColStats, String engine)
> List getPartitionsByNames(String catName, String db_name, String
> tbl_name, List part_names)
> List getPartitionsByNames(String catName, String db_name, String
> tbl_name, List part_names, boolean getColStats, String engine)
> {noformat}
> The problem seems be that every time a new field is added to the request
> object {{GetPartitionsByNamesRequest}} and new variant is introduced in
> IMetastoreClient. Many of these methods are not released yet and it would be
> good to clean them up by using the request object as method argument instead
> of individual fields. Once we release we will not be able to change the
> method signatures since we annotate IMetastoreClient as public API.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25783) Improve rat checking in metastore
[ https://issues.apache.org/jira/browse/HIVE-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-25783: --- Description: In [HIVE-25774|https://issues.apache.org/jira/browse/HIVE-25774], we added ASF license for newly created files in standalone-metastore. The Jira tries to investigate if we can provide some common ways to make sure that the newly added source files contain the ASF license information. (was: The Jira tries to investigate if we can provide rat check to the CI, make sure that the newly added source files contain the ASF license information. ) > Improve rat checking in metastore > -- > > Key: HIVE-25783 > URL: https://issues.apache.org/jira/browse/HIVE-25783 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > In [HIVE-25774|https://issues.apache.org/jira/browse/HIVE-25774], we added > ASF license for newly created files in standalone-metastore. The Jira tries > to investigate if we can provide some common ways to make sure that the newly > added source files contain the ASF license information. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-23971) Cleanup unreleased method signatures in IMetastoreClient
[
https://issues.apache.org/jira/browse/HIVE-23971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ping Xu reassigned HIVE-23971:
--
Assignee: (was: Ping Xu)
> Cleanup unreleased method signatures in IMetastoreClient
>
>
> Key: HIVE-23971
> URL: https://issues.apache.org/jira/browse/HIVE-23971
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Priority: Blocker
>
> There are many methods in IMetastoreClient which are simply wrappers around
> another method. The code has become very intertwined and needs some cleanup.
> For instance, I see the following variations of {{getPartitionsByNames}} in
> {{IMetastoreClient}}
> {noformat}
> List getPartitionsByNames(String db_name, String tbl_name,
> List part_names, boolean getColStats, String engine)
> List getPartitionsByNames(String catName, String db_name, String
> tbl_name, List part_names)
> List getPartitionsByNames(String catName, String db_name, String
> tbl_name, List part_names, boolean getColStats, String engine)
> {noformat}
> The problem seems be that every time a new field is added to the request
> object {{GetPartitionsByNamesRequest}} and new variant is introduced in
> IMetastoreClient. Many of these methods are not released yet and it would be
> good to clean them up by using the request object as method argument instead
> of individual fields. Once we release we will not be able to change the
> method signatures since we annotate IMetastoreClient as public API.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25783) Improve rat checking in standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-25783: --- Summary: Improve rat checking in standalone-metastore (was: Improve rat checking in metastore) > Improve rat checking in standalone-metastore > - > > Key: HIVE-25783 > URL: https://issues.apache.org/jira/browse/HIVE-25783 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > In [HIVE-25774|https://issues.apache.org/jira/browse/HIVE-25774], we added > ASF license for newly created files in standalone-metastore. The Jira tries > to investigate if we can provide some common ways to make sure that the newly > added source files contain the ASF license information. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25783) Improve rat checking in standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-25783: --- Description: In HIVE-25774, we added ASF license for newly created files in standalone-metastore, but we may face the same issue latter on. The Jira tries to investigate if we can provide some common ways to make sure that the newly added source files contain the ASF license information. (was: In [HIVE-25774|https://issues.apache.org/jira/browse/HIVE-25774], we added ASF license for newly created files in standalone-metastore. The Jira tries to investigate if we can provide some common ways to make sure that the newly added source files contain the ASF license information. ) > Improve rat checking in standalone-metastore > - > > Key: HIVE-25783 > URL: https://issues.apache.org/jira/browse/HIVE-25783 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > In HIVE-25774, we added ASF license for newly created files in > standalone-metastore, but we may face the same issue latter on. The Jira > tries to investigate if we can provide some common ways to make sure that the > newly added source files contain the ASF license information. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25783) Improve rat checking in standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-25783?focusedWorklogId=703903&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-703903 ] ASF GitHub Bot logged work on HIVE-25783: - Author: ASF GitHub Bot Created on: 05/Jan/22 12:34 Start Date: 05/Jan/22 12:34 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2852: URL: https://github.com/apache/hive/pull/2852#issuecomment-1005647834 > All of the above points are good. Can we change the jira title/commit message to something which better describes the changes? > > Thanks, Peter Thank you for the review, updated the title. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 703903) Time Spent: 1h 10m (was: 1h) > Improve rat checking in standalone-metastore > - > > Key: HIVE-25783 > URL: https://issues.apache.org/jira/browse/HIVE-25783 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > In HIVE-25774, we added ASF license for newly created files in > standalone-metastore, but we may face the same issue latter on. The Jira > tries to investigate if we can provide some common ways to make sure that the > newly added source files contain the ASF license information. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25841) Improve performance of deleteColumnStatsState
[
https://issues.apache.org/jira/browse/HIVE-25841?focusedWorklogId=703912&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-703912
]
ASF GitHub Bot logged work on HIVE-25841:
-
Author: ASF GitHub Bot
Created on: 05/Jan/22 12:49
Start Date: 05/Jan/22 12:49
Worklog Time Spent: 10m
Work Description: pvary merged pull request #2914:
URL: https://github.com/apache/hive/pull/2914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 703912)
Time Spent: 20m (was: 10m)
> Improve performance of deleteColumnStatsState
> -
>
> Key: HIVE-25841
> URL: https://issues.apache.org/jira/browse/HIVE-25841
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The {{MetaStoreDirectSql.deleteColumnStatsState()}} performance is lacking
> when the {{PARTITION_PARAMS}} and the {{PARTITIONS}} has high number of rows.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25841) Improve performance of deleteColumnStatsState
[
https://issues.apache.org/jira/browse/HIVE-25841?focusedWorklogId=703913&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-703913
]
ASF GitHub Bot logged work on HIVE-25841:
-
Author: ASF GitHub Bot
Created on: 05/Jan/22 12:49
Start Date: 05/Jan/22 12:49
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #2914:
URL: https://github.com/apache/hive/pull/2914#issuecomment-1005658512
Yeah, this was manually tested.
Thanks for the review @kgyrtkirk!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 703913)
Time Spent: 0.5h (was: 20m)
> Improve performance of deleteColumnStatsState
> -
>
> Key: HIVE-25841
> URL: https://issues.apache.org/jira/browse/HIVE-25841
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The {{MetaStoreDirectSql.deleteColumnStatsState()}} performance is lacking
> when the {{PARTITION_PARAMS}} and the {{PARTITIONS}} has high number of rows.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25841) Improve performance of deleteColumnStatsState
[
https://issues.apache.org/jira/browse/HIVE-25841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary resolved HIVE-25841.
---
Fix Version/s: 4.0.0
Resolution: Fixed
Pushed to master.
Thanks for the review [~kgyrtkirk]!
> Improve performance of deleteColumnStatsState
> -
>
> Key: HIVE-25841
> URL: https://issues.apache.org/jira/browse/HIVE-25841
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The {{MetaStoreDirectSql.deleteColumnStatsState()}} performance is lacking
> when the {{PARTITION_PARAMS}} and the {{PARTITIONS}} has high number of rows.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25783) Improve rat checking in standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-25783?focusedWorklogId=703915&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-703915 ] ASF GitHub Bot logged work on HIVE-25783: - Author: ASF GitHub Bot Created on: 05/Jan/22 12:55 Start Date: 05/Jan/22 12:55 Worklog Time Spent: 10m Work Description: pvary commented on pull request #2852: URL: https://github.com/apache/hive/pull/2852#issuecomment-1005662295 > Thank you for the review, updated the title. Thanks @dengzhhu653! Maybe something more generic, like: ``` Clean up standalone-metastore module pom.xml files ``` I think we did plenty of stuff unrelated to rat checks which were for cleaning up the pom.xml by removing unused parts. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 703915) Time Spent: 1h 20m (was: 1h 10m) > Improve rat checking in standalone-metastore > - > > Key: HIVE-25783 > URL: https://issues.apache.org/jira/browse/HIVE-25783 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > In HIVE-25774, we added ASF license for newly created files in > standalone-metastore, but we may face the same issue latter on. The Jira > tries to investigate if we can provide some common ways to make sure that the > newly added source files contain the ASF license information. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25848) Empty result for structs in point lookup optimization with vectorization on
[
https://issues.apache.org/jira/browse/HIVE-25848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ádám Szita updated HIVE-25848:
--
Description:
Repro steps:
{code:java}
set hive.fetch.task.conversion=none;
create table test (a string) partitioned by (y string, m string);
insert into test values ('aa', 2022, 1);
select * from test where (y=year(date_sub(current_date,4)) and
m=month(date_sub(current_date,4))) or (y=year(date_sub(current_date,10)) and
m=month(date_sub(current_date,10)) );
--gives empty result{code}
Turning either of the feature below off yields to good result (1 row expected):
{code:java}
set hive.optimize.point.lookup=false;
set hive.cbo.enable=false;
set hive.vectorized.execution.enabled=false;
{code}
Expected good result is:
{code}
+-+-+-+
| test.a | test.y | test.m |
+-+-+-+
| aa | 2022 | 1 |
+-+-+-+ {code}
> Empty result for structs in point lookup optimization with vectorization on
> ---
>
> Key: HIVE-25848
> URL: https://issues.apache.org/jira/browse/HIVE-25848
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Priority: Major
>
> Repro steps:
> {code:java}
> set hive.fetch.task.conversion=none;
> create table test (a string) partitioned by (y string, m string);
> insert into test values ('aa', 2022, 1);
> select * from test where (y=year(date_sub(current_date,4)) and
> m=month(date_sub(current_date,4))) or (y=year(date_sub(current_date,10)) and
> m=month(date_sub(current_date,10)) );
> --gives empty result{code}
> Turning either of the feature below off yields to good result (1 row
> expected):
> {code:java}
> set hive.optimize.point.lookup=false;
> set hive.cbo.enable=false;
> set hive.vectorized.execution.enabled=false;
> {code}
> Expected good result is:
> {code}
> +-+-+-+
> | test.a | test.y | test.m |
> +-+-+-+
> | aa | 2022 | 1 |
> +-+-+-+ {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25818) Values query with order by position clause fails
[
https://issues.apache.org/jira/browse/HIVE-25818?focusedWorklogId=704003&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-704003
]
ASF GitHub Bot logged work on HIVE-25818:
-
Author: ASF GitHub Bot
Created on: 05/Jan/22 14:24
Start Date: 05/Jan/22 14:24
Worklog Time Spent: 10m
Work Description: kasakrisz opened a new pull request #2922:
URL: https://github.com/apache/hive/pull/2922
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 704003)
Time Spent: 1h 50m (was: 1h 40m)
> Values query with order by position clause fails
>
>
> Key: HIVE-25818
> URL: https://issues.apache.org/jira/browse/HIVE-25818
> Project: Hive
> Issue Type: Bug
> Components: CBO, Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> {code}
> values(1+1, 2, 5.0, 'a') order by 1 limit 2;
> {code}
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.getFieldIndexFromColumnNumber(CalcitePlanner.java:4146)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.beginGenOBLogicalPlan(CalcitePlanner.java:4028)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genOBLogicalPlan(CalcitePlanner.java:3933)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5148)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1651)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1593)
> at
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1345)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:563)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12565)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:456)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:317)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:105)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:500)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:453)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:417)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:411)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:256)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:422)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:353)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:726)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:696)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:114)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
> at
> org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver(TestMiniLlapLocalCliDriver.java:62)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ
[jira] [Work logged] (HIVE-25843) Add flag to disable Iceberg FileIO config serialization
[ https://issues.apache.org/jira/browse/HIVE-25843?focusedWorklogId=704018&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-704018 ] ASF GitHub Bot logged work on HIVE-25843: - Author: ASF GitHub Bot Created on: 05/Jan/22 14:38 Start Date: 05/Jan/22 14:38 Worklog Time Spent: 10m Work Description: marton-bod commented on pull request #2917: URL: https://github.com/apache/hive/pull/2917#issuecomment-1005740910 @pvary Can you please take an initial look? I'm still thinking about the best way to do this, but currently I think using a validation method on the storage handler is the best way to go. Not entirely comfortable with tying this new method to the FileSinkDesc (ideally I'd like to make it a bit more generic) but so far that was the only thing that worked out well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 704018) Time Spent: 20m (was: 10m) > Add flag to disable Iceberg FileIO config serialization > --- > > Key: HIVE-25843 > URL: https://issues.apache.org/jira/browse/HIVE-25843 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Hive serializes the Iceberg table object into each individual split. Since > the FileIO is part of the Iceberg table and it has its own hadoop > configuration, this configuration will be the dominant factor determining the > size of the serialized split. In our tests we have found that due to this > serialized config, iceberg splits are 15-20x larger than normal Hive splits > (which led to OOM in some of our perf tests). > This PR proposes to introduce a config which can turn off this config > serialization, and let the deserializer-side fill out the config values > instead (which works for Hive executors, since they have all the config > values in hand). This can reduce the Iceberg split size by ~20x based on > local tests. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25843) Add flag to disable Iceberg FileIO config serialization
[ https://issues.apache.org/jira/browse/HIVE-25843?focusedWorklogId=704019&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-704019 ] ASF GitHub Bot logged work on HIVE-25843: - Author: ASF GitHub Bot Created on: 05/Jan/22 14:38 Start Date: 05/Jan/22 14:38 Worklog Time Spent: 10m Work Description: marton-bod removed a comment on pull request #2917: URL: https://github.com/apache/hive/pull/2917#issuecomment-1005740910 @pvary Can you please take an initial look? I'm still thinking about the best way to do this, but currently I think using a validation method on the storage handler is the best way to go. Not entirely comfortable with tying this new method to the FileSinkDesc (ideally I'd like to make it a bit more generic) but so far that was the only thing that worked out well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 704019) Time Spent: 0.5h (was: 20m) > Add flag to disable Iceberg FileIO config serialization > --- > > Key: HIVE-25843 > URL: https://issues.apache.org/jira/browse/HIVE-25843 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Hive serializes the Iceberg table object into each individual split. Since > the FileIO is part of the Iceberg table and it has its own hadoop > configuration, this configuration will be the dominant factor determining the > size of the serialized split. In our tests we have found that due to this > serialized config, iceberg splits are 15-20x larger than normal Hive splits > (which led to OOM in some of our perf tests). > This PR proposes to introduce a config which can turn off this config > serialization, and let the deserializer-side fill out the config values > instead (which works for Hive executors, since they have all the config > values in hand). This can reduce the Iceberg split size by ~20x based on > local tests. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25783) Improve rat checking in standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-25783?focusedWorklogId=704021&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-704021 ] ASF GitHub Bot logged work on HIVE-25783: - Author: ASF GitHub Bot Created on: 05/Jan/22 14:41 Start Date: 05/Jan/22 14:41 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2852: URL: https://github.com/apache/hive/pull/2852#issuecomment-1005743620 > > Thank you for the review, updated the title. > > Thanks @dengzhhu653! Maybe something more generic, like: > > ``` > Clean up standalone-metastore module pom.xml files > ``` > > I think we did plenty of stuff unrelated to rat checks which were for cleaning up the pom.xml by removing unused parts. > > What do you think? The `Clean up pom files` is part of what we want to do in this pull request, though I have no good idea, maybe `Refine standalone-metastore module pom.xml files`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 704021) Time Spent: 1.5h (was: 1h 20m) > Improve rat checking in standalone-metastore > - > > Key: HIVE-25783 > URL: https://issues.apache.org/jira/browse/HIVE-25783 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > In HIVE-25774, we added ASF license for newly created files in > standalone-metastore, but we may face the same issue latter on. The Jira > tries to investigate if we can provide some common ways to make sure that the > newly added source files contain the ASF license information. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25783) Improve rat checking in standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-25783?focusedWorklogId=704030&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-704030 ] ASF GitHub Bot logged work on HIVE-25783: - Author: ASF GitHub Bot Created on: 05/Jan/22 14:59 Start Date: 05/Jan/22 14:59 Worklog Time Spent: 10m Work Description: pvary commented on pull request #2852: URL: https://github.com/apache/hive/pull/2852#issuecomment-1005759135 > Refine standalone-metastore module pom.xml files +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 704030) Time Spent: 1h 40m (was: 1.5h) > Improve rat checking in standalone-metastore > - > > Key: HIVE-25783 > URL: https://issues.apache.org/jira/browse/HIVE-25783 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > In HIVE-25774, we added ASF license for newly created files in > standalone-metastore, but we may face the same issue latter on. The Jira > tries to investigate if we can provide some common ways to make sure that the > newly added source files contain the ASF license information. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25783) Refine standalone-metastore module pom.xml files
[ https://issues.apache.org/jira/browse/HIVE-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-25783: --- Summary: Refine standalone-metastore module pom.xml files (was: Improve rat checking in standalone-metastore) > Refine standalone-metastore module pom.xml files > > > Key: HIVE-25783 > URL: https://issues.apache.org/jira/browse/HIVE-25783 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > In HIVE-25774, we added ASF license for newly created files in > standalone-metastore, but we may face the same issue latter on. The Jira > tries to investigate if we can provide some common ways to make sure that the > newly added source files contain the ASF license information. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25275) OOM during query planning due to HiveJoinPushTransitivePredicatesRule matching infinitely
[
https://issues.apache.org/jira/browse/HIVE-25275?focusedWorklogId=704133&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-704133
]
ASF GitHub Bot logged work on HIVE-25275:
-
Author: ASF GitHub Bot
Created on: 05/Jan/22 17:55
Start Date: 05/Jan/22 17:55
Worklog Time Spent: 10m
Work Description: asolimando opened a new pull request #2923:
URL: https://github.com/apache/hive/pull/2923
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 704133)
Time Spent: 0.5h (was: 20m)
> OOM during query planning due to HiveJoinPushTransitivePredicatesRule
> matching infinitely
> -
>
> Key: HIVE-25275
> URL: https://issues.apache.org/jira/browse/HIVE-25275
> Project: Hive
> Issue Type: Bug
>Reporter: László Pintér
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> While running the following query OOM is raised during the planning phase
> {code:sql}
> CREATE TABLE A (`value_date` date) STORED AS ORC;
> CREATE TABLE B (`business_date` date) STORED AS ORC;
> SELECT A.VALUE_DATE
> FROM A, B
> WHERE A.VALUE_DATE = BUSINESS_DATE
> AND A.VALUE_DATE = TRUNC(BUSINESS_DATE, 'MONTH');
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25707) SchemaTool may leave the metastore in-between upgrade steps
[ https://issues.apache.org/jira/browse/HIVE-25707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469482#comment-17469482 ] Rahul Pathak commented on HIVE-25707: - This is also useful if SchemaTool upgrade fails for some data inconsistency, like duplicate records in tables NEXT_**_ID Rerunning SchemaTool upgrade in such case create more issues. There should be way to put back the state as it was before starting the upgrade steps > SchemaTool may leave the metastore in-between upgrade steps > --- > > Key: HIVE-25707 > URL: https://issues.apache.org/jira/browse/HIVE-25707 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > > it seems like: > * schematool runs the sql files via beeline > * autocommit is turned on > * pressing ctrl+c or killing the process will result in an invalid schema > https://github.com/apache/hive/blob/6e02f6164385a370ee8014c795bee1fa423d7937/beeline/src/java/org/apache/hive/beeline/schematool/HiveSchemaTool.java#L79 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25663) Need to modify table/partition lock acquisition retry for Zookeeper lock manager
[
https://issues.apache.org/jira/browse/HIVE-25663?focusedWorklogId=704342&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-704342
]
ASF GitHub Bot logged work on HIVE-25663:
-
Author: ASF GitHub Bot
Created on: 06/Jan/22 00:13
Start Date: 06/Jan/22 00:13
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #2761:
URL: https://github.com/apache/hive/pull/2761
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 704342)
Time Spent: 0.5h (was: 20m)
> Need to modify table/partition lock acquisition retry for Zookeeper lock
> manager
>
>
> Key: HIVE-25663
> URL: https://issues.apache.org/jira/browse/HIVE-25663
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Eugene Chung
>Assignee: Eugene Chung
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-10-30-11-54-42-164.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> LOCK TABLE default.my_table PARTITION (log_date='2021-10-30') EXCLUSIVE;
> SET hive.query.timeout.seconds=5;
> SELECT * FROM default.my_table WHERE log_date='2021-10-30' LIMIT 10;
> {code}
> If you execute the three SQLs above in the same session, the last SELECT will
> be cancelled by timeout error. The problem is that when you execute 'show
> locks', you will see a SHARED lock of default.my_table which is remained for
> 100 minutes, if you are using ZooKeeperHiveLockManager.
> !image-2021-10-30-11-54-42-164.png|width=873,height=411!
> I am going to explain the problem one by one.
>
> The SELECT SQL which gets some data from a partitioned table
> {code:java}
> SELECT * FROM my_table WHERE log_date='2021-10-30' LIMIT 10{code}
> needs two SHARED locks in order. The two SHARED locks are
> * default.my_table
> * default.my_table@log_date=2021-10-30
> Before executing the SQL, an EXCLUSIVE lock of the partition exists. We can
> simulate it easily with a DDL like below;
> {code:java}
> LOCK TABLE default.my_table PARTITION (log_date='2021-10-30') EXCLUSIVE{code}
>
> The SELECT SQL acquires the SHARED lock of the table, but it can't acquire
> the SHARED lock of the partition. It retries to acquire it as specified by
> two configurations. The default values mean it will retry for 100 minutes.
> * hive.lock.sleep.between.retries=60s
> * hive.lock.numretries=100
>
> If query.timeout is set to 5 seconds, the SELECT SQL is cancelled 5 seconds
> later and the client returns with timeout error. But the SHARED lock of the
> my_table is still remained for 100 minutes, because [the current
> ZooKeeperHiveLockManager just logs
> InterruptedException|https://github.com/apache/hive/blob/8a8e03d02003aa3543f46f595b4425fd8c156ad9/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java#L326]
> and still goes on lock retry. This also means that the SQL processing thread
> is still doing its job for 100 minutes even though the SQL is cancelled. If
> the same SQL is executed 3 times, you can see 3 threads each of which thread
> dump is like below;
> {code:java}
> "HiveServer2-Background-Pool: Thread-154" #154 prio=5 os_prio=0
> tid=0x7f0ac91cb000 nid=0x13d25 waiting on condition [0x000
> 07f0aa2ce2000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.lock(ZooKeeperHiveLockManager.java:303)
> at
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.lock(ZooKeeperHiveLockManager.java:207)
> at
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager.acquireLocks(DummyTxnManager.java:199)
> at org.apache.hadoop.hive.ql.Driver.acquireLocks(Driver.java:1610)
> at org.apache.hadoop.hive.ql.Driver.lockAndRespond(Driver.java:1796)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1966)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1710)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1704)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:217)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$500(SQLOperation.java:87)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOpera
[jira] [Work logged] (HIVE-25826) Support table defaults at each database level
[ https://issues.apache.org/jira/browse/HIVE-25826?focusedWorklogId=704357&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-704357 ] ASF GitHub Bot logged work on HIVE-25826: - Author: ASF GitHub Bot Created on: 06/Jan/22 00:56 Start Date: 06/Jan/22 00:56 Worklog Time Spent: 10m Work Description: saihemanth-cloudera opened a new pull request #2924: URL: https://github.com/apache/hive/pull/2924 ### What changes were proposed in this pull request? Introduced a new feature to support table default types at each database level ### Why are the changes needed? Users now would be able to specify the default table type at database level. Previously setting the default table type is only possible from hive-site.xml or at table level. ### Does this PR introduce _any_ user-facing change? Yes. Users would need to create/alter database with the database property "defaultTableType"="EXTERNAL" ### How was this patch tested? Local machine, remote cluster, Added unit test cases -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 704357) Remaining Estimate: 0h Time Spent: 10m > Support table defaults at each database level > - > > Key: HIVE-25826 > URL: https://issues.apache.org/jira/browse/HIVE-25826 > Project: Hive > Issue Type: New Feature > Components: HiveServer2, Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This feature jira is for adding support for users being able to specify > default table types at each database level. Currently, this is configurable > at the service level (HS2) and at each JDBC session-level. The goal is to be > able to specify the default table type for each database either when the DB > is created (create DB DDL) or at any point later (via alter db DDL). > More details and design docs for this feature will be added soon to this Jira. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25826) Support table defaults at each database level
[ https://issues.apache.org/jira/browse/HIVE-25826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25826: -- Labels: pull-request-available (was: ) > Support table defaults at each database level > - > > Key: HIVE-25826 > URL: https://issues.apache.org/jira/browse/HIVE-25826 > Project: Hive > Issue Type: New Feature > Components: HiveServer2, Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This feature jira is for adding support for users being able to specify > default table types at each database level. Currently, this is configurable > at the service level (HS2) and at each JDBC session-level. The goal is to be > able to specify the default table type for each database either when the DB > is created (create DB DDL) or at any point later (via alter db DDL). > More details and design docs for this feature will be added soon to this Jira. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25846) Ensure that deregistering hive servers works, even after zookeeper session expired
[ https://issues.apache.org/jira/browse/HIVE-25846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeongdae Kim updated HIVE-25846: Description: Zookeeper watchers are one time trigger and when zookeeper session is expired by long gc pause or something and is reconnected, all watchers already registered are gone, and deregister mechanism doesn't work. we should register deregister watchers again to get notification. (was: Zookeeper watchers are one time trigger and when zookeeper session is expired by long gc pause or something and is reconnected, all watchers already registered are gone. so, we should add deregister watchers again to get notification.) > Ensure that deregistering hive servers works, even after zookeeper session > expired > -- > > Key: HIVE-25846 > URL: https://issues.apache.org/jira/browse/HIVE-25846 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.8, 3.1.2 >Reporter: Jeongdae Kim >Assignee: Jeongdae Kim >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Zookeeper watchers are one time trigger and when zookeeper session is expired > by long gc pause or something and is reconnected, all watchers already > registered are gone, and deregister mechanism doesn't work. we should > register deregister watchers again to get notification. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25846) Ensure that deregistering hive servers works, even after zookeeper session expired
[ https://issues.apache.org/jira/browse/HIVE-25846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeongdae Kim updated HIVE-25846: Description: Zookeeper watchers are one time trigger and when zookeeper session is expired by long gc pause or something and is reconnected, all watchers already registered are gone, and deregister mechanism doesn't work. We should register deregister watchers again to get notification, after session re-established was:Zookeeper watchers are one time trigger and when zookeeper session is expired by long gc pause or something and is reconnected, all watchers already registered are gone, and deregister mechanism doesn't work. we should register deregister watchers again to get notification. > Ensure that deregistering hive servers works, even after zookeeper session > expired > -- > > Key: HIVE-25846 > URL: https://issues.apache.org/jira/browse/HIVE-25846 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.8, 3.1.2 >Reporter: Jeongdae Kim >Assignee: Jeongdae Kim >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Zookeeper watchers are one time trigger and when zookeeper session is expired > by long gc pause or something and is reconnected, all watchers already > registered are gone, and deregister mechanism doesn't work. > We should register deregister watchers again to get notification, after > session re-established -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25846) Ensure that deregistering hive servers works, even after zookeeper session expired
[ https://issues.apache.org/jira/browse/HIVE-25846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeongdae Kim updated HIVE-25846: Description: Zookeeper watchers are one time trigger and when zookeeper session is expired by long gc pause or something and is reconnected, all watchers already registered are gone, and deregister mechanism doesn't work. We should register deregister watchers again to get notification, after session re-established. was: Zookeeper watchers are one time trigger and when zookeeper session is expired by long gc pause or something and is reconnected, all watchers already registered are gone, and deregister mechanism doesn't work. We should register deregister watchers again to get notification, after session re-established > Ensure that deregistering hive servers works, even after zookeeper session > expired > -- > > Key: HIVE-25846 > URL: https://issues.apache.org/jira/browse/HIVE-25846 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.8, 3.1.2 >Reporter: Jeongdae Kim >Assignee: Jeongdae Kim >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Zookeeper watchers are one time trigger and when zookeeper session is expired > by long gc pause or something and is reconnected, all watchers already > registered are gone, and deregister mechanism doesn't work. > We should register deregister watchers again to get notification, after > session re-established. -- This message was sent by Atlassian Jira (v8.20.1#820001)
