[jira] [Commented] (HIVE-26813) Upgrade HikariCP from 2.6.1 to 4.0.3.
[ https://issues.apache.org/jira/browse/HIVE-26813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644450#comment-17644450 ] Dagang Wei commented on HIVE-26813: --- +1 > Upgrade HikariCP from 2.6.1 to 4.0.3. > - > > Key: HIVE-26813 > URL: https://issues.apache.org/jira/browse/HIVE-26813 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The Hive Metastore currently integrates with HikariCP 2.6.1 for database > connection pooling. This version was released in 2017. The most recent Java > 8-compatible release is 4.0.3, released earlier this year. This bug proposes > to upgrade so that we can include the past few years of development and bug > fixes in the 4.0.0 GA release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27485) Backport HIVE-22170 to branch-3
Dagang Wei created HIVE-27485: - Summary: Backport HIVE-22170 to branch-3 Key: HIVE-27485 URL: https://issues.apache.org/jira/browse/HIVE-27485 Project: Hive Issue Type: Sub-task Affects Versions: 3.1.3 Reporter: Dagang Wei Assignee: Dagang Wei -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27485) Backport HIVE-22170 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dagang Wei updated HIVE-27485: -- Target Version/s: 3.2.0 (was: 3.1.4) > Backport HIVE-22170 to branch-3 > --- > > Key: HIVE-27485 > URL: https://issues.apache.org/jira/browse/HIVE-27485 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.3 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27485) Backport HIVE-22170 (unix timestamp) to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dagang Wei updated HIVE-27485: -- Summary: Backport HIVE-22170 (unix timestamp) to branch-3 (was: Backport HIVE-22170 to branch-3) > Backport HIVE-22170 (unix timestamp) to branch-3 > > > Key: HIVE-27485 > URL: https://issues.apache.org/jira/browse/HIVE-27485 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.3 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27485) Backport HIVE-22170 (unix timestamp) to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dagang Wei updated HIVE-27485: -- Description: https://issues.apache.org/jira/browse/HIVE-22170 and https://issues.apache.org/jira/browse/HIVE-22331 > Backport HIVE-22170 (unix timestamp) to branch-3 > > > Key: HIVE-27485 > URL: https://issues.apache.org/jira/browse/HIVE-27485 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.3 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Blocker > > https://issues.apache.org/jira/browse/HIVE-22170 and > https://issues.apache.org/jira/browse/HIVE-22331 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-20600) Metastore connection leak
[
https://issues.apache.org/jira/browse/HIVE-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153084#comment-17153084
]
Dagang Wei commented on HIVE-20600:
---
nemonlou, I'm looking for the patch for 2.3.3, do you know if the patch
attached in this issue solves the problem?
https://issues.apache.org/jira/secure/attachment/12940453/HIVE-20600.patch
> Metastore connection leak
> -
>
> Key: HIVE-20600
> URL: https://issues.apache.org/jira/browse/HIVE-20600
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.3
>Reporter: Damon Cortesi
>Priority: Major
> Attachments: HIVE-20600.patch, consume_threads.py
>
>
> Within the execute method of HiveServer2, there appears to be a connection
> leak. With fairly straightforward series of INSERT statements, the connection
> count in the logs continues to increase over time. Under certain loads, this
> can also consume all underlying threads of the Hive metastore and result in
> HS2 becoming unresponsive to new connections.
> The log below is the result of some python code executing a single insert
> statement, and then looping through a series of 10 more insert statements. We
> can see there's one dangling connection left open after each execution
> leaving us with 12 open connections (11 from the execute statements + 1 from
> HS2 startup).
> {code}
> 2018-09-19T17:14:32,108 INFO [main([])]: hive.metastore
> (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore,
> current connections: 1
> 2018-09-19T17:14:48,175 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b
> HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore
> (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore,
> current connections: 2
> 2018-09-19T17:15:05,543 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 1
> 2018-09-19T17:15:05,548 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 2
> 2018-09-19T17:15:05,932 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 1
> 2018-09-19T17:15:05,935 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 2
> 2018-09-19T17:15:06,123 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 1
> 2018-09-19T17:15:06,126 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 2
> ...
> 2018-09-19T17:15:20,626 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b
> HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore
> (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore,
> current connections: 12
> 2018-09-19T17:15:21,153 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 11
> 2018-09-19T17:15:21,155 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 12
> 2018-09-19T17:15:21,306 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 11
> 2018-09-19T17:15:21,308 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 12
> 2018-09-19T17:15:21,385 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 11
> 2018-09-19T17:15:21,387 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 12
> 2018-09-19T17:15:21,541 INFO [HiveServer2-Handler-Pool: Thread-31([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 13
> 2018-09-19T17:15:21,542 INFO [HiveServer2-Handler-Pool: Thread-31([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 12
> {code}
> Attached is a simple [impyla|https://github.com/cloude
[jira] [Comment Edited] (HIVE-20600) Metastore connection leak
[
https://issues.apache.org/jira/browse/HIVE-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153084#comment-17153084
]
Dagang Wei edited comment on HIVE-20600 at 7/7/20, 8:42 PM:
[~nemonlou], I'm looking for the patch for 2.3.3, do you know if the patch
attached in this issue solves the problem? [^HIVE-20600.patch]
was (Author: functicons):
nemonlou, I'm looking for the patch for 2.3.3, do you know if the patch
attached in this issue solves the problem?
https://issues.apache.org/jira/secure/attachment/12940453/HIVE-20600.patch
> Metastore connection leak
> -
>
> Key: HIVE-20600
> URL: https://issues.apache.org/jira/browse/HIVE-20600
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.3
>Reporter: Damon Cortesi
>Priority: Major
> Attachments: HIVE-20600.patch, consume_threads.py
>
>
> Within the execute method of HiveServer2, there appears to be a connection
> leak. With fairly straightforward series of INSERT statements, the connection
> count in the logs continues to increase over time. Under certain loads, this
> can also consume all underlying threads of the Hive metastore and result in
> HS2 becoming unresponsive to new connections.
> The log below is the result of some python code executing a single insert
> statement, and then looping through a series of 10 more insert statements. We
> can see there's one dangling connection left open after each execution
> leaving us with 12 open connections (11 from the execute statements + 1 from
> HS2 startup).
> {code}
> 2018-09-19T17:14:32,108 INFO [main([])]: hive.metastore
> (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore,
> current connections: 1
> 2018-09-19T17:14:48,175 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b
> HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore
> (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore,
> current connections: 2
> 2018-09-19T17:15:05,543 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 1
> 2018-09-19T17:15:05,548 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 2
> 2018-09-19T17:15:05,932 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 1
> 2018-09-19T17:15:05,935 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 2
> 2018-09-19T17:15:06,123 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 1
> 2018-09-19T17:15:06,126 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 2
> ...
> 2018-09-19T17:15:20,626 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b
> HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore
> (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore,
> current connections: 12
> 2018-09-19T17:15:21,153 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 11
> 2018-09-19T17:15:21,155 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 12
> 2018-09-19T17:15:21,306 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 11
> 2018-09-19T17:15:21,308 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 12
> 2018-09-19T17:15:21,385 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 11
> 2018-09-19T17:15:21,387 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 12
> 2018-09-19T17:15:21,541 INFO [HiveServer2-Handler-Pool: Thread-31([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 13
> 2018-09-19T17:15:21,542 INFO [HiveServer
[jira] [Commented] (HIVE-20693) Case-sensitivity for column names when reading from ORC
[
https://issues.apache.org/jira/browse/HIVE-20693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17492847#comment-17492847
]
Dagang Wei commented on HIVE-20693:
---
The workaround is mentioned here in the mail archive [1]:
{{set orc.force.positional.evolution=true;}}{{}}
[1]:
[https://www.mail-archive.com/user@orc.apache.org/msg00229.html|https://www.mail-archive.com/user@orc.apache.org/msg00229.html.]
> Case-sensitivity for column names when reading from ORC
> ---
>
> Key: HIVE-20693
> URL: https://issues.apache.org/jira/browse/HIVE-20693
> Project: Hive
> Issue Type: Bug
> Components: Hive, ORC
>Affects Versions: 2.3.2
>Reporter: Alexandre Crayssac
>Priority: Major
>
> Hello everyone,
> I observed a different behavior between version 1.2.1 and 2.3.2 (that's the
> only two versions I've been able to test).
> When creating an external table pointing to ORC files and having upper cased
> column names in the ORC files metadata I'm able to read the data on 1.2.1 but
> not on 2.3.2.
> I tested with both upper cased and lower cased column names in my CREATE
> TABLE statement and it does not work in both cases. Looks like normal since
> column names are normalized to lower case in Hive.
> So, I would like to know if this is a feature or a bug in Hive 2.3.2 ?
> In fact, if this is a feature it would be impossible to have upper case
> column names in ORC files with Hive 2.3.2.
> Please, let me know if you need more informations.
> Kind regards,
> Alexandre
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (HIVE-20693) Case-sensitivity for column names when reading from ORC
[
https://issues.apache.org/jira/browse/HIVE-20693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17492847#comment-17492847
]
Dagang Wei edited comment on HIVE-20693 at 2/15/22, 7:39 PM:
-
The workaround is mentioned here in the mail archive [1]:
{{set orc.force.positional.evolution=true;}}
[1]:
[https://www.mail-archive.com/user@orc.apache.org/msg00229.html|https://www.mail-archive.com/user@orc.apache.org/msg00229.html.]
was (Author: functicons):
The workaround is mentioned here in the mail archive [1]:
{{set orc.force.positional.evolution=true;}}{{}}
[1]:
[https://www.mail-archive.com/user@orc.apache.org/msg00229.html|https://www.mail-archive.com/user@orc.apache.org/msg00229.html.]
> Case-sensitivity for column names when reading from ORC
> ---
>
> Key: HIVE-20693
> URL: https://issues.apache.org/jira/browse/HIVE-20693
> Project: Hive
> Issue Type: Bug
> Components: Hive, ORC
>Affects Versions: 2.3.2
>Reporter: Alexandre Crayssac
>Priority: Major
>
> Hello everyone,
> I observed a different behavior between version 1.2.1 and 2.3.2 (that's the
> only two versions I've been able to test).
> When creating an external table pointing to ORC files and having upper cased
> column names in the ORC files metadata I'm able to read the data on 1.2.1 but
> not on 2.3.2.
> I tested with both upper cased and lower cased column names in my CREATE
> TABLE statement and it does not work in both cases. Looks like normal since
> column names are normalized to lower case in Hive.
> So, I would like to know if this is a feature or a bug in Hive 2.3.2 ?
> In fact, if this is a feature it would be impossible to have upper case
> column names in ORC files with Hive 2.3.2.
> Please, let me know if you need more informations.
> Kind regards,
> Alexandre
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-20784) Migrate hbase.util.Base64 to java.util.Base64
[ https://issues.apache.org/jira/browse/HIVE-20784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dagang Wei reassigned HIVE-20784: - > Migrate hbase.util.Base64 to java.util.Base64 > - > > Key: HIVE-20784 > URL: https://issues.apache.org/jira/browse/HIVE-20784 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 3.1.0 > Environment: HBase 2.0.2 > Hive 3.1.0 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Critical > > By default Hive 3.1.0 depends on HBase 2.0.0-alpha4. HBase 2.0.2 migrated > from hbase.util.Base64 to java.util.Base64 (HBASE-20884), which causes Hive > 3.1.0 fails to build with HBase 2.0.2. > > $ cd hbase-handler > $ mvn package -DskipTests -Dhbase.version=2.0.2 > [ERROR] > .../hive/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[29,36] > cannot find symbol > [ERROR] symbol: class Base64 > [ERROR] location: package org.apache.hadoop.hbase.util > > To make Hive works with 2.0.2+ (and also older versions), we should consider > migrating Hive to java.util.Base64. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15326) Hive shims report Unrecognized Hadoop major version number: 3.0.0-alpha2-SNAPSHOT
[
https://issues.apache.org/jira/browse/HIVE-15326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662986#comment-16662986
]
Dagang Wei commented on HIVE-15326:
---
Is it possible to backport it to Hive 1.2? I assume it would help resolve
https://issues.apache.org/jira/browse/SPARK-18673.
> Hive shims report Unrecognized Hadoop major version number:
> 3.0.0-alpha2-SNAPSHOT
> -
>
> Key: HIVE-15326
> URL: https://issues.apache.org/jira/browse/HIVE-15326
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 1.2.1
> Environment: Hadoop trunk branch
>Reporter: Steve Loughran
>Priority: Major
> Fix For: 3.0.0
>
>
> Hive built against Hadoop 2 fails to run against Hadoop 3.x,
> declaring:{{Unrecognized Hadoop major version number: 3.0.0-alpha2-SNAPSHOT}}
> Refusing to play on Hadoop 3.x may actually be the correct behaviour, though
> ideally we've retained API compatibility to everything works (maybe with some
> CP tweaking).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20784) Migrate hbase.util.Base64 to java.util.Base64
[ https://issues.apache.org/jira/browse/HIVE-20784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dagang Wei reassigned HIVE-20784: - Assignee: Dagang Wei > Migrate hbase.util.Base64 to java.util.Base64 > - > > Key: HIVE-20784 > URL: https://issues.apache.org/jira/browse/HIVE-20784 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 3.1.0 > Environment: HBase 2.0.2 > Hive 3.1.0 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Critical > Labels: pull-request-available > > By default Hive 3.1.0 depends on HBase 2.0.0-alpha4. HBase 2.0.2 migrated > from hbase.util.Base64 to java.util.Base64 (HBASE-20884), which causes Hive > 3.1.0 fails to build with HBase 2.0.2. > > $ cd hbase-handler > $ mvn package -DskipTests -Dhbase.version=2.0.2 > [ERROR] > .../hive/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[29,36] > cannot find symbol > [ERROR] symbol: class Base64 > [ERROR] location: package org.apache.hadoop.hbase.util > > To make Hive works with 2.0.2+ (and also older versions), we should consider > migrating Hive to java.util.Base64. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
