[jira] [Commented] (HIVE-26813) Upgrade HikariCP from 2.6.1 to 4.0.3.
[ https://issues.apache.org/jira/browse/HIVE-26813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644450#comment-17644450 ] Dagang Wei commented on HIVE-26813: --- +1 > Upgrade HikariCP from 2.6.1 to 4.0.3. > - > > Key: HIVE-26813 > URL: https://issues.apache.org/jira/browse/HIVE-26813 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The Hive Metastore currently integrates with HikariCP 2.6.1 for database > connection pooling. This version was released in 2017. The most recent Java > 8-compatible release is 4.0.3, released earlier this year. This bug proposes > to upgrade so that we can include the past few years of development and bug > fixes in the 4.0.0 GA release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27485) Backport HIVE-22170 to branch-3
Dagang Wei created HIVE-27485: - Summary: Backport HIVE-22170 to branch-3 Key: HIVE-27485 URL: https://issues.apache.org/jira/browse/HIVE-27485 Project: Hive Issue Type: Sub-task Affects Versions: 3.1.3 Reporter: Dagang Wei Assignee: Dagang Wei -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27485) Backport HIVE-22170 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dagang Wei updated HIVE-27485: -- Target Version/s: 3.2.0 (was: 3.1.4) > Backport HIVE-22170 to branch-3 > --- > > Key: HIVE-27485 > URL: https://issues.apache.org/jira/browse/HIVE-27485 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.3 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27485) Backport HIVE-22170 (unix timestamp) to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dagang Wei updated HIVE-27485: -- Summary: Backport HIVE-22170 (unix timestamp) to branch-3 (was: Backport HIVE-22170 to branch-3) > Backport HIVE-22170 (unix timestamp) to branch-3 > > > Key: HIVE-27485 > URL: https://issues.apache.org/jira/browse/HIVE-27485 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.3 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27485) Backport HIVE-22170 (unix timestamp) to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dagang Wei updated HIVE-27485: -- Description: https://issues.apache.org/jira/browse/HIVE-22170 and https://issues.apache.org/jira/browse/HIVE-22331 > Backport HIVE-22170 (unix timestamp) to branch-3 > > > Key: HIVE-27485 > URL: https://issues.apache.org/jira/browse/HIVE-27485 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.3 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Blocker > > https://issues.apache.org/jira/browse/HIVE-22170 and > https://issues.apache.org/jira/browse/HIVE-22331 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-20600) Metastore connection leak
[ https://issues.apache.org/jira/browse/HIVE-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153084#comment-17153084 ] Dagang Wei commented on HIVE-20600: --- nemonlou, I'm looking for the patch for 2.3.3, do you know if the patch attached in this issue solves the problem? https://issues.apache.org/jira/secure/attachment/12940453/HIVE-20600.patch > Metastore connection leak > - > > Key: HIVE-20600 > URL: https://issues.apache.org/jira/browse/HIVE-20600 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.3.3 >Reporter: Damon Cortesi >Priority: Major > Attachments: HIVE-20600.patch, consume_threads.py > > > Within the execute method of HiveServer2, there appears to be a connection > leak. With fairly straightforward series of INSERT statements, the connection > count in the logs continues to increase over time. Under certain loads, this > can also consume all underlying threads of the Hive metastore and result in > HS2 becoming unresponsive to new connections. > The log below is the result of some python code executing a single insert > statement, and then looping through a series of 10 more insert statements. We > can see there's one dangling connection left open after each execution > leaving us with 12 open connections (11 from the execute statements + 1 from > HS2 startup). > {code} > 2018-09-19T17:14:32,108 INFO [main([])]: hive.metastore > (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, > current connections: 1 > 2018-09-19T17:14:48,175 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b > HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore > (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, > current connections: 2 > 2018-09-19T17:15:05,543 INFO [HiveServer2-Background-Pool: Thread-36([])]: > hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to > metastore, current connections: 1 > 2018-09-19T17:15:05,548 INFO [HiveServer2-Background-Pool: Thread-36([])]: > hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to > metastore, current connections: 2 > 2018-09-19T17:15:05,932 INFO [HiveServer2-Background-Pool: Thread-36([])]: > hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to > metastore, current connections: 1 > 2018-09-19T17:15:05,935 INFO [HiveServer2-Background-Pool: Thread-36([])]: > hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to > metastore, current connections: 2 > 2018-09-19T17:15:06,123 INFO [HiveServer2-Background-Pool: Thread-36([])]: > hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to > metastore, current connections: 1 > 2018-09-19T17:15:06,126 INFO [HiveServer2-Background-Pool: Thread-36([])]: > hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to > metastore, current connections: 2 > ... > 2018-09-19T17:15:20,626 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b > HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore > (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, > current connections: 12 > 2018-09-19T17:15:21,153 INFO [HiveServer2-Background-Pool: Thread-162([])]: > hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to > metastore, current connections: 11 > 2018-09-19T17:15:21,155 INFO [HiveServer2-Background-Pool: Thread-162([])]: > hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to > metastore, current connections: 12 > 2018-09-19T17:15:21,306 INFO [HiveServer2-Background-Pool: Thread-162([])]: > hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to > metastore, current connections: 11 > 2018-09-19T17:15:21,308 INFO [HiveServer2-Background-Pool: Thread-162([])]: > hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to > metastore, current connections: 12 > 2018-09-19T17:15:21,385 INFO [HiveServer2-Background-Pool: Thread-162([])]: > hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to > metastore, current connections: 11 > 2018-09-19T17:15:21,387 INFO [HiveServer2-Background-Pool: Thread-162([])]: > hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to > metastore, current connections: 12 > 2018-09-19T17:15:21,541 INFO [HiveServer2-Handler-Pool: Thread-31([])]: > hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to > metastore, current connections: 13 > 2018-09-19T17:15:21,542 INFO [HiveServer2-Handler-Pool: Thread-31([])]: > hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to > metastore, current connections: 12 > {code} > Attached is a simple [impyla|https://github.com/cloude
[jira] [Comment Edited] (HIVE-20600) Metastore connection leak
[ https://issues.apache.org/jira/browse/HIVE-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153084#comment-17153084 ] Dagang Wei edited comment on HIVE-20600 at 7/7/20, 8:42 PM: [~nemonlou], I'm looking for the patch for 2.3.3, do you know if the patch attached in this issue solves the problem? [^HIVE-20600.patch] was (Author: functicons): nemonlou, I'm looking for the patch for 2.3.3, do you know if the patch attached in this issue solves the problem? https://issues.apache.org/jira/secure/attachment/12940453/HIVE-20600.patch > Metastore connection leak > - > > Key: HIVE-20600 > URL: https://issues.apache.org/jira/browse/HIVE-20600 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.3.3 >Reporter: Damon Cortesi >Priority: Major > Attachments: HIVE-20600.patch, consume_threads.py > > > Within the execute method of HiveServer2, there appears to be a connection > leak. With fairly straightforward series of INSERT statements, the connection > count in the logs continues to increase over time. Under certain loads, this > can also consume all underlying threads of the Hive metastore and result in > HS2 becoming unresponsive to new connections. > The log below is the result of some python code executing a single insert > statement, and then looping through a series of 10 more insert statements. We > can see there's one dangling connection left open after each execution > leaving us with 12 open connections (11 from the execute statements + 1 from > HS2 startup). > {code} > 2018-09-19T17:14:32,108 INFO [main([])]: hive.metastore > (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, > current connections: 1 > 2018-09-19T17:14:48,175 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b > HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore > (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, > current connections: 2 > 2018-09-19T17:15:05,543 INFO [HiveServer2-Background-Pool: Thread-36([])]: > hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to > metastore, current connections: 1 > 2018-09-19T17:15:05,548 INFO [HiveServer2-Background-Pool: Thread-36([])]: > hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to > metastore, current connections: 2 > 2018-09-19T17:15:05,932 INFO [HiveServer2-Background-Pool: Thread-36([])]: > hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to > metastore, current connections: 1 > 2018-09-19T17:15:05,935 INFO [HiveServer2-Background-Pool: Thread-36([])]: > hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to > metastore, current connections: 2 > 2018-09-19T17:15:06,123 INFO [HiveServer2-Background-Pool: Thread-36([])]: > hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to > metastore, current connections: 1 > 2018-09-19T17:15:06,126 INFO [HiveServer2-Background-Pool: Thread-36([])]: > hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to > metastore, current connections: 2 > ... > 2018-09-19T17:15:20,626 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b > HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore > (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, > current connections: 12 > 2018-09-19T17:15:21,153 INFO [HiveServer2-Background-Pool: Thread-162([])]: > hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to > metastore, current connections: 11 > 2018-09-19T17:15:21,155 INFO [HiveServer2-Background-Pool: Thread-162([])]: > hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to > metastore, current connections: 12 > 2018-09-19T17:15:21,306 INFO [HiveServer2-Background-Pool: Thread-162([])]: > hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to > metastore, current connections: 11 > 2018-09-19T17:15:21,308 INFO [HiveServer2-Background-Pool: Thread-162([])]: > hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to > metastore, current connections: 12 > 2018-09-19T17:15:21,385 INFO [HiveServer2-Background-Pool: Thread-162([])]: > hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to > metastore, current connections: 11 > 2018-09-19T17:15:21,387 INFO [HiveServer2-Background-Pool: Thread-162([])]: > hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to > metastore, current connections: 12 > 2018-09-19T17:15:21,541 INFO [HiveServer2-Handler-Pool: Thread-31([])]: > hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to > metastore, current connections: 13 > 2018-09-19T17:15:21,542 INFO [HiveServer
[jira] [Commented] (HIVE-20693) Case-sensitivity for column names when reading from ORC
[ https://issues.apache.org/jira/browse/HIVE-20693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17492847#comment-17492847 ] Dagang Wei commented on HIVE-20693: --- The workaround is mentioned here in the mail archive [1]: {{set orc.force.positional.evolution=true;}}{{}} [1]: [https://www.mail-archive.com/user@orc.apache.org/msg00229.html|https://www.mail-archive.com/user@orc.apache.org/msg00229.html.] > Case-sensitivity for column names when reading from ORC > --- > > Key: HIVE-20693 > URL: https://issues.apache.org/jira/browse/HIVE-20693 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Affects Versions: 2.3.2 >Reporter: Alexandre Crayssac >Priority: Major > > Hello everyone, > I observed a different behavior between version 1.2.1 and 2.3.2 (that's the > only two versions I've been able to test). > When creating an external table pointing to ORC files and having upper cased > column names in the ORC files metadata I'm able to read the data on 1.2.1 but > not on 2.3.2. > I tested with both upper cased and lower cased column names in my CREATE > TABLE statement and it does not work in both cases. Looks like normal since > column names are normalized to lower case in Hive. > So, I would like to know if this is a feature or a bug in Hive 2.3.2 ? > In fact, if this is a feature it would be impossible to have upper case > column names in ORC files with Hive 2.3.2. > Please, let me know if you need more informations. > Kind regards, > Alexandre -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (HIVE-20693) Case-sensitivity for column names when reading from ORC
[ https://issues.apache.org/jira/browse/HIVE-20693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17492847#comment-17492847 ] Dagang Wei edited comment on HIVE-20693 at 2/15/22, 7:39 PM: - The workaround is mentioned here in the mail archive [1]: {{set orc.force.positional.evolution=true;}} [1]: [https://www.mail-archive.com/user@orc.apache.org/msg00229.html|https://www.mail-archive.com/user@orc.apache.org/msg00229.html.] was (Author: functicons): The workaround is mentioned here in the mail archive [1]: {{set orc.force.positional.evolution=true;}}{{}} [1]: [https://www.mail-archive.com/user@orc.apache.org/msg00229.html|https://www.mail-archive.com/user@orc.apache.org/msg00229.html.] > Case-sensitivity for column names when reading from ORC > --- > > Key: HIVE-20693 > URL: https://issues.apache.org/jira/browse/HIVE-20693 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Affects Versions: 2.3.2 >Reporter: Alexandre Crayssac >Priority: Major > > Hello everyone, > I observed a different behavior between version 1.2.1 and 2.3.2 (that's the > only two versions I've been able to test). > When creating an external table pointing to ORC files and having upper cased > column names in the ORC files metadata I'm able to read the data on 1.2.1 but > not on 2.3.2. > I tested with both upper cased and lower cased column names in my CREATE > TABLE statement and it does not work in both cases. Looks like normal since > column names are normalized to lower case in Hive. > So, I would like to know if this is a feature or a bug in Hive 2.3.2 ? > In fact, if this is a feature it would be impossible to have upper case > column names in ORC files with Hive 2.3.2. > Please, let me know if you need more informations. > Kind regards, > Alexandre -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-20784) Migrate hbase.util.Base64 to java.util.Base64
[ https://issues.apache.org/jira/browse/HIVE-20784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dagang Wei reassigned HIVE-20784: - > Migrate hbase.util.Base64 to java.util.Base64 > - > > Key: HIVE-20784 > URL: https://issues.apache.org/jira/browse/HIVE-20784 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 3.1.0 > Environment: HBase 2.0.2 > Hive 3.1.0 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Critical > > By default Hive 3.1.0 depends on HBase 2.0.0-alpha4. HBase 2.0.2 migrated > from hbase.util.Base64 to java.util.Base64 (HBASE-20884), which causes Hive > 3.1.0 fails to build with HBase 2.0.2. > > $ cd hbase-handler > $ mvn package -DskipTests -Dhbase.version=2.0.2 > [ERROR] > .../hive/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[29,36] > cannot find symbol > [ERROR] symbol: class Base64 > [ERROR] location: package org.apache.hadoop.hbase.util > > To make Hive works with 2.0.2+ (and also older versions), we should consider > migrating Hive to java.util.Base64. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15326) Hive shims report Unrecognized Hadoop major version number: 3.0.0-alpha2-SNAPSHOT
[ https://issues.apache.org/jira/browse/HIVE-15326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662986#comment-16662986 ] Dagang Wei commented on HIVE-15326: --- Is it possible to backport it to Hive 1.2? I assume it would help resolve https://issues.apache.org/jira/browse/SPARK-18673. > Hive shims report Unrecognized Hadoop major version number: > 3.0.0-alpha2-SNAPSHOT > - > > Key: HIVE-15326 > URL: https://issues.apache.org/jira/browse/HIVE-15326 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 1.2.1 > Environment: Hadoop trunk branch >Reporter: Steve Loughran >Priority: Major > Fix For: 3.0.0 > > > Hive built against Hadoop 2 fails to run against Hadoop 3.x, > declaring:{{Unrecognized Hadoop major version number: 3.0.0-alpha2-SNAPSHOT}} > Refusing to play on Hadoop 3.x may actually be the correct behaviour, though > ideally we've retained API compatibility to everything works (maybe with some > CP tweaking). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20784) Migrate hbase.util.Base64 to java.util.Base64
[ https://issues.apache.org/jira/browse/HIVE-20784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dagang Wei reassigned HIVE-20784: - Assignee: Dagang Wei > Migrate hbase.util.Base64 to java.util.Base64 > - > > Key: HIVE-20784 > URL: https://issues.apache.org/jira/browse/HIVE-20784 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 3.1.0 > Environment: HBase 2.0.2 > Hive 3.1.0 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Critical > Labels: pull-request-available > > By default Hive 3.1.0 depends on HBase 2.0.0-alpha4. HBase 2.0.2 migrated > from hbase.util.Base64 to java.util.Base64 (HBASE-20884), which causes Hive > 3.1.0 fails to build with HBase 2.0.2. > > $ cd hbase-handler > $ mvn package -DskipTests -Dhbase.version=2.0.2 > [ERROR] > .../hive/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[29,36] > cannot find symbol > [ERROR] symbol: class Base64 > [ERROR] location: package org.apache.hadoop.hbase.util > > To make Hive works with 2.0.2+ (and also older versions), we should consider > migrating Hive to java.util.Base64. -- This message was sent by Atlassian JIRA (v7.6.3#76005)