[jira] [Created] (HIVE-27457) Update Kudu table storage handlers in backend db
Zoltan Chovan created HIVE-27457: Summary: Update Kudu table storage handlers in backend db Key: HIVE-27457 URL: https://issues.apache.org/jira/browse/HIVE-27457 Project: Hive Issue Type: Improvement Reporter: Zoltan Chovan Assignee: Zoltan Chovan Before https://issues.apache.org/jira/browse/KUDU-3401 the following table parameters were not filled out when creating a Kudu backed Hive table: SerDe Library InputFormat OutputFormat This change is adding the necessary update statements to the db schema ddls so that existing Kudu tables can be used after a schema update. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-25442) Initiator speed-up: only read compaction history once per loop
[ https://issues.apache.org/jira/browse/HIVE-25442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-25442: Assignee: (was: Zoltan Chovan) > Initiator speed-up: only read compaction history once per loop > -- > > Key: HIVE-25442 > URL: https://issues.apache.org/jira/browse/HIVE-25442 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Priority: Major > > In checkFailedCompactions (which is called for every partition in the list of > potentials to compact) we select from metadata table COMPLETED_COMPACTIONS. > But the Initiator main loop already has a ShowCompactResponse. We can use > that instead. > For cases where metadata is huge, this will help speed up the Initiator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24445) Non blocking DROP table implementation
[ https://issues.apache.org/jira/browse/HIVE-24445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-24445: Assignee: (was: Zoltan Chovan) > Non blocking DROP table implementation > -- > > Key: HIVE-24445 > URL: https://issues.apache.org/jira/browse/HIVE-24445 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Zoltan Chovan >Priority: Major > Labels: pull-request-available > Time Spent: 4h 20m > Remaining Estimate: 0h > > Implement a way to execute drop table operations in a way that doesn't have > to wait for currently running read operations to be finished. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22628) Add locks and transactions tables from sys db to information_schema
[ https://issues.apache.org/jira/browse/HIVE-22628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-22628: Assignee: (was: Zoltan Chovan) > Add locks and transactions tables from sys db to information_schema > --- > > Key: HIVE-22628 > URL: https://issues.apache.org/jira/browse/HIVE-22628 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24798) refactor TxnHandler.cleanupRecords to use predefined query strings
[ https://issues.apache.org/jira/browse/HIVE-24798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan resolved HIVE-24798. -- Resolution: Abandoned > refactor TxnHandler.cleanupRecords to use predefined query strings > -- > > Key: HIVE-24798 > URL: https://issues.apache.org/jira/browse/HIVE-24798 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > TxnHandler.cleanupRecords should use predefined query strings instead of > using a stringbuffer to build the delete queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25074) Remove Metastore flushCache usage
[
https://issues.apache.org/jira/browse/HIVE-25074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan reassigned HIVE-25074:
Assignee: (was: Zoltan Chovan)
> Remove Metastore flushCache usage
> -
>
> Key: HIVE-25074
> URL: https://issues.apache.org/jira/browse/HIVE-25074
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Miklos Szurap
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The "flushCache" in HiveMetaStore with the ObjectStore implementation is
> currently a NOOP:
> {code:java}
> public void flushCache() {
> // NOP as there's no caching
> } {code}
> The HBaseStore (HBaseReadWrite) had some logic in it, however it has been
> removed in HIVE-17234.
> As I see the calls are going like this:
> HiveMetaStoreClient.flushCache() -> CachedStore.flushCache() ->
> ObjectStore.flushCache()
> There are significant amount of calls (about 10% of all calls) made from the
> client to the server - to do nothing. We could spare the call to the server
> completely, including getting a DB connection which can take 1+ seconds under
> high load scenarios slowing down Hive queries unnecessarily.
> Can we:
> # Deprecate the RawStore.flushCache (if there are other implementations)
> # Deprecate the HiveMetaStoreClient.flushCache()
> # Do the NOOP on the client side in HiveMetaStoreClient.flushCache() (while
> it is not removed in a next version)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-23391: Assignee: (was: Zoltan Chovan) > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23391.2.patch, HIVE-23391.3.patch, > HIVE-23391.4.patch, HIVE-23391.patch > > Time Spent: 40m > Remaining Estimate: 0h > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25346) cleanTxnToWriteIdTable breaks SNAPSHOT isolation
[ https://issues.apache.org/jira/browse/HIVE-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-25346: Assignee: Denys Kuzmenko (was: Zoltan Chovan) > cleanTxnToWriteIdTable breaks SNAPSHOT isolation > > > Key: HIVE-25346 > URL: https://issues.apache.org/jira/browse/HIVE-25346 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Chovan >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 11h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24753) Non blocking DROP PARTITION implementation
[ https://issues.apache.org/jira/browse/HIVE-24753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-24753: Assignee: (was: Zoltan Chovan) > Non blocking DROP PARTITION implementation > -- > > Key: HIVE-24753 > URL: https://issues.apache.org/jira/browse/HIVE-24753 > Project: Hive > Issue Type: New Feature >Reporter: Zoltan Chovan >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Implement a way to execute drop partition operations in a way that doesn't > have to wait for currently running read operations to be finished. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25442) Initiator speed-up: only read compaction history once per loop
[ https://issues.apache.org/jira/browse/HIVE-25442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-25442: > Initiator speed-up: only read compaction history once per loop > -- > > Key: HIVE-25442 > URL: https://issues.apache.org/jira/browse/HIVE-25442 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > > In checkFailedCompactions (which is called for every partition in the list of > potentials to compact) we select from metadata table COMPLETED_COMPACTIONS. > But the Initiator main loop already has a ShowCompactResponse. We can use > that instead. > For cases where metadata is huge, this will help speed up the Initiator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25346) cleanTxnToWriteIdTable breaks SNAPSHOT isolation
[ https://issues.apache.org/jira/browse/HIVE-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-25346: > cleanTxnToWriteIdTable breaks SNAPSHOT isolation > > > Key: HIVE-25346 > URL: https://issues.apache.org/jira/browse/HIVE-25346 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25074) Remove Metastore flushCache usage
[
https://issues.apache.org/jira/browse/HIVE-25074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-25074:
-
Labels: (was: perf)
> Remove Metastore flushCache usage
> -
>
> Key: HIVE-25074
> URL: https://issues.apache.org/jira/browse/HIVE-25074
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Miklos Szurap
>Priority: Major
>
> The "flushCache" in HiveMetaStore with the ObjectStore implementation is
> currently a NOOP:
> {code:java}
> public void flushCache() {
> // NOP as there's no caching
> } {code}
> The HBaseStore (HBaseReadWrite) had some logic in it, however it has been
> removed in HIVE-17234.
> As I see the calls are going like this:
> HiveMetaStoreClient.flushCache() -> CachedStore.flushCache() ->
> ObjectStore.flushCache()
> There are significant amount of calls (about 10% of all calls) made from the
> client to the server - to do nothing. We could spare the call to the server
> completely, including getting a DB connection which can take 1+ seconds under
> high load scenarios slowing down Hive queries unnecessarily.
> Can we:
> # Deprecate the RawStore.flushCache (if there are other implementations)
> # Deprecate the HiveMetaStoreClient.flushCache()
> # Do the NOOP on the client side in HiveMetaStoreClient.flushCache() (while
> it is not removed in a next version)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-25074) Remove Metastore flushCache usage
[
https://issues.apache.org/jira/browse/HIVE-25074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan reassigned HIVE-25074:
Assignee: Zoltan Chovan
> Remove Metastore flushCache usage
> -
>
> Key: HIVE-25074
> URL: https://issues.apache.org/jira/browse/HIVE-25074
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Miklos Szurap
>Assignee: Zoltan Chovan
>Priority: Major
>
> The "flushCache" in HiveMetaStore with the ObjectStore implementation is
> currently a NOOP:
> {code:java}
> public void flushCache() {
> // NOP as there's no caching
> } {code}
> The HBaseStore (HBaseReadWrite) had some logic in it, however it has been
> removed in HIVE-17234.
> As I see the calls are going like this:
> HiveMetaStoreClient.flushCache() -> CachedStore.flushCache() ->
> ObjectStore.flushCache()
> There are significant amount of calls (about 10% of all calls) made from the
> client to the server - to do nothing. We could spare the call to the server
> completely, including getting a DB connection which can take 1+ seconds under
> high load scenarios slowing down Hive queries unnecessarily.
> Can we:
> # Deprecate the RawStore.flushCache (if there are other implementations)
> # Deprecate the HiveMetaStoreClient.flushCache()
> # Do the NOOP on the client side in HiveMetaStoreClient.flushCache() (while
> it is not removed in a next version)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25074) Remove Metastore flushCache usage
[
https://issues.apache.org/jira/browse/HIVE-25074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-25074:
-
Labels: perf (was: )
> Remove Metastore flushCache usage
> -
>
> Key: HIVE-25074
> URL: https://issues.apache.org/jira/browse/HIVE-25074
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Miklos Szurap
>Priority: Major
> Labels: perf
>
> The "flushCache" in HiveMetaStore with the ObjectStore implementation is
> currently a NOOP:
> {code:java}
> public void flushCache() {
> // NOP as there's no caching
> } {code}
> The HBaseStore (HBaseReadWrite) had some logic in it, however it has been
> removed in HIVE-17234.
> As I see the calls are going like this:
> HiveMetaStoreClient.flushCache() -> CachedStore.flushCache() ->
> ObjectStore.flushCache()
> There are significant amount of calls (about 10% of all calls) made from the
> client to the server - to do nothing. We could spare the call to the server
> completely, including getting a DB connection which can take 1+ seconds under
> high load scenarios slowing down Hive queries unnecessarily.
> Can we:
> # Deprecate the RawStore.flushCache (if there are other implementations)
> # Deprecate the HiveMetaStoreClient.flushCache()
> # Do the NOOP on the client side in HiveMetaStoreClient.flushCache() (while
> it is not removed in a next version)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24862) Fix race condition causing NPE during dynamic partition loading
[
https://issues.apache.org/jira/browse/HIVE-24862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan reassigned HIVE-24862:
Assignee: Zoltan Chovan
> Fix race condition causing NPE during dynamic partition loading
> ---
>
> Key: HIVE-24862
> URL: https://issues.apache.org/jira/browse/HIVE-24862
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Zoltan Chovan
>Priority: Major
>
> Following properties default to 15 threads.
> {noformat}
> hive.load.dynamic.partitions.thread
> hive.mv.files.thread
> {noformat}
> During loadDynamicPartitions, it ends ups initializing {{newFiles}} without
> synchronization (HIVE-20661, HIVE-24738).
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2871]
> This causes race condition when dynamic partition thread internally makes use
> of {{hive.mv.files.threads}} in copyFiles/replaceFiles.
> This causes "NPE" during retrieval in {{addInsertFileInformation()}}.
>
> e.g stacktrace
> {noformat}
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.fs.FileSystem.fixRelativePart(FileSystem.java:2734)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.fixRelativePart(DistributedFileSystem.java:3396)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1740)
> at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1740)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.addInsertFileInformation(Hive.java:3566)
> at org.apache.hadoop.hive.ql.metadata.Hive.fireInsertEvent(Hive.java:3540)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2414)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$4(Hive.java:2909)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24798) refactor TxnHandler.cleanupRecords to use predefined query strings
[ https://issues.apache.org/jira/browse/HIVE-24798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-24798: > refactor TxnHandler.cleanupRecords to use predefined query strings > -- > > Key: HIVE-24798 > URL: https://issues.apache.org/jira/browse/HIVE-24798 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Minor > > TxnHandler.cleanupRecords should use predefined query strings instead of > using a stringbuffer to build the delete queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24753) Non blocking DROP PARTITION implementation
[ https://issues.apache.org/jira/browse/HIVE-24753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-24753: Assignee: Zoltan Chovan > Non blocking DROP PARTITION implementation > -- > > Key: HIVE-24753 > URL: https://issues.apache.org/jira/browse/HIVE-24753 > Project: Hive > Issue Type: New Feature >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > > Implement a way to execute drop partition operations in a way that doesn't > have to wait for currently running read operations to be finished. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24445) Non blocking DROP table implementation
[ https://issues.apache.org/jira/browse/HIVE-24445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-24445: > Non blocking DROP table implementation > -- > > Key: HIVE-24445 > URL: https://issues.apache.org/jira/browse/HIVE-24445 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > > Implement a way to execute drop table operations in a way that doesn't have > to wait for currently running read operations to be finished. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Attachment: HIVE-22869.9.patch > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, > HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, > HIVE-22869.7.patch, HIVE-22869.8.patch, HIVE-22869.9.patch, HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Attachment: HIVE-22869.8.patch > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, > HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, > HIVE-22869.7.patch, HIVE-22869.8.patch, HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Status: Open (was: Patch Available) > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, > HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, > HIVE-22869.7.patch, HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Status: Patch Available (was: Open) > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, > HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, > HIVE-22869.7.patch, HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Attachment: HIVE-22869.7.patch > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, > HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, > HIVE-22869.7.patch, HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23391: - Status: Open (was: Patch Available) > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23391.2.patch, HIVE-23391.3.patch, > HIVE-23391.4.patch, HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23391: - Attachment: HIVE-23391.4.patch > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23391.2.patch, HIVE-23391.3.patch, > HIVE-23391.4.patch, HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23391: - Status: Patch Available (was: Open) > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23391.2.patch, HIVE-23391.3.patch, > HIVE-23391.4.patch, HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23391: - Status: Patch Available (was: In Progress) > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23391.2.patch, HIVE-23391.3.patch, HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23391: - Status: In Progress (was: Patch Available) > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23391.2.patch, HIVE-23391.3.patch, HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17108313#comment-17108313 ] Zoltan Chovan commented on HIVE-23391: -- [~dkuzmenko] could you review this? > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23391.2.patch, HIVE-23391.3.patch, HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23391: - Attachment: HIVE-23391.3.patch > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23391.2.patch, HIVE-23391.3.patch, HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Attachment: HIVE-22869.6.patch > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, > HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Attachment: HIVE-22869.5.patch > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, > HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23391: - Attachment: HIVE-23391.2.patch > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23391.2.patch, HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23414) Detail Hive Java Compatibility
[ https://issues.apache.org/jira/browse/HIVE-23414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102621#comment-17102621 ] Zoltan Chovan commented on HIVE-23414: -- Seems right to me, to be honest I was under the impression that the current upstream master already supported Java 11, but then I saw https://issues.apache.org/jira/browse/HIVE-22415 > Detail Hive Java Compatibility > -- > > Key: HIVE-23414 > URL: https://issues.apache.org/jira/browse/HIVE-23414 > Project: Hive > Issue Type: Improvement > Components: Documentation >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Trivial > Attachments: HIVE-23414.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23391: - Attachment: HIVE-23391.patch > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-23391: Assignee: Zoltan Chovan > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23391 started by Zoltan Chovan. > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23391: - Fix Version/s: 4.0.0 Status: Patch Available (was: In Progress) > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23363: - Attachment: HIVE-23363.2.patch > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23363.2.patch, HIVE-23363.patch > > > Upgrade Datanucleus from 4.2 to 5.2 as based on it's docs 4.2 has been > retired: > [http://www.datanucleus.org/documentation/products.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23363: - Description: Upgrade Datanucleus from 4.2 to 5.2 as based on it's docs 4.2 has been retired: [http://www.datanucleus.org/documentation/products.html] > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23363.patch > > > Upgrade Datanucleus from 4.2 to 5.2 as based on it's docs 4.2 has been > retired: > [http://www.datanucleus.org/documentation/products.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23363 started by Zoltan Chovan. > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23363.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23363: - Attachment: HIVE-23363.patch > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23363.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-23363: Assignee: Zoltan Chovan > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23363.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23363: - Affects Version/s: 4.0.0 Status: Patch Available (was: In Progress) > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23363.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Attachment: HIVE-22869.4.patch > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, > HIVE-22869.4.patch, HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23243) Accept SQL type like pattern for Show Databases
[ https://issues.apache.org/jira/browse/HIVE-23243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087505#comment-17087505 ] Zoltan Chovan commented on HIVE-23243: -- [~dmollitor] FYI > Accept SQL type like pattern for Show Databases > --- > > Key: HIVE-23243 > URL: https://issues.apache.org/jira/browse/HIVE-23243 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Minor > Attachments: HIVE-23243.01.patch > > > Show Databases pattern accepts java like pattern with * and ., use SQL like > instead with % and _. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Attachment: HIVE-22869.3.patch > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23126) Add TXNS table to sysdb
[ https://issues.apache.org/jira/browse/HIVE-23126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073467#comment-17073467 ] Zoltan Chovan commented on HIVE-23126: -- [~rameshkumar] I've already done this in https://issues.apache.org/jira/browse/HIVE-22553 > Add TXNS table to sysdb > --- > > Key: HIVE-23126 > URL: https://issues.apache.org/jira/browse/HIVE-23126 > Project: Hive > Issue Type: Task >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23126.1.patch, HIVE-23126.2.patch > > > Add TXNS table to sysdb -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Attachment: HIVE-22869.2.patch > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.2.patch, HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23044) Make sure Cleaner doesn't delete delta directories for running queries
[ https://issues.apache.org/jira/browse/HIVE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064930#comment-17064930 ] Zoltan Chovan commented on HIVE-23044: -- Out of the 141 test, 10 fail the rest timeouts. The failing tests fail without the patch applied as well. > Make sure Cleaner doesn't delete delta directories for running queries > -- > > Key: HIVE-23044 > URL: https://issues.apache.org/jira/browse/HIVE-23044 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23044.1.branch-3.1.patch, > HIVE-23044.2.branch-3.1.patch, HIVE-23044.3.branch-3.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23044) Make sure Cleaner doesn't delete delta directories for running queries
[ https://issues.apache.org/jira/browse/HIVE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23044: - Attachment: HIVE-23044.3.branch-3.1.patch > Make sure Cleaner doesn't delete delta directories for running queries > -- > > Key: HIVE-23044 > URL: https://issues.apache.org/jira/browse/HIVE-23044 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23044.1.branch-3.1.patch, > HIVE-23044.2.branch-3.1.patch, HIVE-23044.3.branch-3.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23044) Make sure Cleaner doesn't delete delta directories for running queries
[ https://issues.apache.org/jira/browse/HIVE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23044: - Attachment: HIVE-23044.2.branch-3.1.patch > Make sure Cleaner doesn't delete delta directories for running queries > -- > > Key: HIVE-23044 > URL: https://issues.apache.org/jira/browse/HIVE-23044 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23044.1.branch-3.1.patch, > HIVE-23044.2.branch-3.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HIVE-23044) Make sure Cleaner doesn't delete delta directories for running queries
[ https://issues.apache.org/jira/browse/HIVE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23044: - Comment: was deleted (was: uploading empty patchfile to establish baseline) > Make sure Cleaner doesn't delete delta directories for running queries > -- > > Key: HIVE-23044 > URL: https://issues.apache.org/jira/browse/HIVE-23044 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23044.1.branch-3.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23044) Make sure Cleaner doesn't delete delta directories for running queries
[ https://issues.apache.org/jira/browse/HIVE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061887#comment-17061887 ] Zoltan Chovan commented on HIVE-23044: -- uploading empty patchfile to establish baseline > Make sure Cleaner doesn't delete delta directories for running queries > -- > > Key: HIVE-23044 > URL: https://issues.apache.org/jira/browse/HIVE-23044 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23044.1.branch-3.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23044) Make sure Cleaner doesn't delete delta directories for running queries
[ https://issues.apache.org/jira/browse/HIVE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23044: - Attachment: (was: HIVE-23044.branch-3.1.1.patch) > Make sure Cleaner doesn't delete delta directories for running queries > -- > > Key: HIVE-23044 > URL: https://issues.apache.org/jira/browse/HIVE-23044 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23044.1.branch-3.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23044) Make sure Cleaner doesn't delete delta directories for running queries
[ https://issues.apache.org/jira/browse/HIVE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23044: - Attachment: HIVE-23044.1.branch-3.1.patch > Make sure Cleaner doesn't delete delta directories for running queries > -- > > Key: HIVE-23044 > URL: https://issues.apache.org/jira/browse/HIVE-23044 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23044.1.branch-3.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23044) Make sure Cleaner doesn't delete delta directories for running queries
[ https://issues.apache.org/jira/browse/HIVE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23044: - Status: Patch Available (was: In Progress) > Make sure Cleaner doesn't delete delta directories for running queries > -- > > Key: HIVE-23044 > URL: https://issues.apache.org/jira/browse/HIVE-23044 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23044.1.branch-3.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23044) Make sure Cleaner doesn't delete delta directories for running queries
[ https://issues.apache.org/jira/browse/HIVE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23044: - Attachment: HIVE-23044.branch-3.1.1.patch > Make sure Cleaner doesn't delete delta directories for running queries > -- > > Key: HIVE-23044 > URL: https://issues.apache.org/jira/browse/HIVE-23044 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23044.branch-3.1.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23044) Make sure Cleaner doesn't delete delta directories for running queries
[ https://issues.apache.org/jira/browse/HIVE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23044: - Attachment: (was: HIVE-23044.patch) > Make sure Cleaner doesn't delete delta directories for running queries > -- > > Key: HIVE-23044 > URL: https://issues.apache.org/jira/browse/HIVE-23044 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23044) Make sure Cleaner doesn't delete delta directories for running queries
[ https://issues.apache.org/jira/browse/HIVE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23044: - Attachment: HIVE-23044.patch > Make sure Cleaner doesn't delete delta directories for running queries > -- > > Key: HIVE-23044 > URL: https://issues.apache.org/jira/browse/HIVE-23044 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23044) Make sure Cleaner doesn't delete delta directories for running queries
[ https://issues.apache.org/jira/browse/HIVE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23044 started by Zoltan Chovan. > Make sure Cleaner doesn't delete delta directories for running queries > -- > > Key: HIVE-23044 > URL: https://issues.apache.org/jira/browse/HIVE-23044 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23044) Make sure Cleaner doesn't delete delta directories for running queries
[ https://issues.apache.org/jira/browse/HIVE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-23044: > Make sure Cleaner doesn't delete delta directories for running queries > -- > > Key: HIVE-23044 > URL: https://issues.apache.org/jira/browse/HIVE-23044 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22942) Replace PTest with an alternative
[ https://issues.apache.org/jira/browse/HIVE-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17055892#comment-17055892 ] Zoltan Chovan commented on HIVE-22942: -- So far it sounds great :) What happens when a build fails for a PR? If there are flaky tests, we would need some way to re-trigger the run. Would it be possible to only execute the failed tests for the same commit? If we already using k8s, do you think it would be feasible to introduce multiple types of backend dbs for certain tests? We have a lot of directsql around TxnHandler and HMS, that would greatly benefit from that. The backend db could be spun up on-demand just for some tests that are flagged maybe. > Replace PTest with an alternative > - > > Key: HIVE-22942 > URL: https://issues.apache.org/jira/browse/HIVE-22942 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > I never opened a jira about this...but it might actually help collect ideas > and actually start going somewhere sooner than later :D > Right now we maintain the ptest2 project inside Hive to be able to run Hive > tests in a distributed fashion...the backstab of this solution is that we are > putting much effort into maintaining a distributed test execution framework... > I think it would be better if we could find an off the shelf solution for the > task and migrate to that instead of putting more efforts into the ptest > framework -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17050028#comment-17050028
]
Zoltan Chovan commented on HIVE-22899:
--
[~pvary] could you review?
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.3.patch,
> HIVE-22899.4.patch, HIVE-22899.5.patch, HIVE-22899.6.patch,
> HIVE-22899.7.patch, HIVE-22899.8.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22899:
-
Attachment: HIVE-22899.8.patch
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.3.patch,
> HIVE-22899.4.patch, HIVE-22899.5.patch, HIVE-22899.6.patch,
> HIVE-22899.7.patch, HIVE-22899.8.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22899:
-
Attachment: HIVE-22899.7.patch
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.3.patch,
> HIVE-22899.4.patch, HIVE-22899.5.patch, HIVE-22899.6.patch,
> HIVE-22899.7.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22899:
-
Attachment: HIVE-22899.6.patch
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.3.patch,
> HIVE-22899.4.patch, HIVE-22899.5.patch, HIVE-22899.6.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22899:
-
Attachment: HIVE-22899.5.patch
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.3.patch,
> HIVE-22899.4.patch, HIVE-22899.5.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22899:
-
Attachment: (was: HIVE-22899.4.patch)
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.3.patch,
> HIVE-22899.4.patch, HIVE-22899.5.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22899:
-
Attachment: HIVE-22899.4.patch
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.3.patch,
> HIVE-22899.4.patch, HIVE-22899.4.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22899:
-
Attachment: HIVE-22899.4.patch
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.3.patch,
> HIVE-22899.4.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22899:
-
Attachment: HIVE-22899.3.patch
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.3.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17040080#comment-17040080
]
Zoltan Chovan edited comment on HIVE-22899 at 2/19/20 2:10 PM:
---
[~pvary] turns out that this specific test (rfc5424_parser_file_pruning.q) is
excluded[1] from the tests (note, it still shows up in the batches, but then
I'm assuming it gets ignored by the TestCliDriver). I'll remove my changes
regarding both the q and out files.
[1]
[https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java#L254|https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java#L254]
was (Author: zchovan):
[~pvary] turns out that this specific test (rfc5424_parser_file_pruning.q) is
excluded from the tests (note, it still shows up in the batches, but then I'm
assumign it gets ignored by the TestCliDriver). I'll remove my changes
regarding both the q and out files.
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17040080#comment-17040080
]
Zoltan Chovan commented on HIVE-22899:
--
[~pvary] turns out that this specific test (rfc5424_parser_file_pruning.q) is
excluded from the tests (note, it still shows up in the batches, but then I'm
assumign it gets ignored by the TestCliDriver). I'll remove my changes
regarding both the q and out files.
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17040022#comment-17040022
]
Zoltan Chovan commented on HIVE-22899:
--
[~pvary]
Regarding the lines "TOTAL_TABLE_ROWS_WRITTEN" and "CREATED_FILES" in
rfc5424_parser_file_pruning.q.out I wasn't sure either. I've run the following
in the latest upstream master branch:
{code:java}
mvn test -Dtest=TestMiniLlapLocalCliDriver
-Dqfile=rfc5424_parser_file_pruning.q
{code}
and it fails. Not sure if this shows up in current precommits or not.
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22899:
-
Attachment: HIVE-22899.2.patch
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.2.patch, HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22904) Compaction cleaner cannot find COMPACTION_QUEUE table using postgres db
[
https://issues.apache.org/jira/browse/HIVE-22904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039813#comment-17039813
]
Zoltan Chovan commented on HIVE-22904:
--
Make sure that cq_id is uppercased as well
> Compaction cleaner cannot find COMPACTION_QUEUE table using postgres db
> ---
>
> Key: HIVE-22904
> URL: https://issues.apache.org/jira/browse/HIVE-22904
> Project: Hive
> Issue Type: Bug
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Attachments: HIVE-22904.01.patch, HIVE-22904.02.patch
>
>
> In CompactionTxnHandler
> {code:java}
> delete from COMPACTION_QUEUE where cq_id = ?
> {code}
> fails with
> {code:java}
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not exist
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22899:
-
Status: Patch Available (was: Open)
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22899:
-
Attachment: HIVE-22899.patch
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-22899.patch
>
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22899) Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan reassigned HIVE-22899:
> Make sure qtests clean up copied files from test directories
>
>
> Key: HIVE-22899
> URL: https://issues.apache.org/jira/browse/HIVE-22899
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
>
> Several qtest files are copying schema or test files to the test directories
> (such as ${system:test.tmp.dir} and
> ${hiveconf:hive.metastore.warehouse.dir}), many times without changing the
> name of the copied file. When the same files is copied by another qtest to
> the same directory the copy and hence the test fails. This can lead to flaky
> tests when any two of these qtests gets scheduled to the same batch.
>
> In order to avoid these failures, we should make sure the files copied to the
> test dirs have unique names and we should make sure these files are cleaned
> up by the same qtest files that copies the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22850) Optimise lock acquisition in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035216#comment-17035216
]
Zoltan Chovan commented on HIVE-22850:
--
[~rajesh.balamohan] is there a reviewboard or pr for this change?
> Optimise lock acquisition in TxnHandler
> ---
>
> Key: HIVE-22850
> URL: https://issues.apache.org/jira/browse/HIVE-22850
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-22850.1.patch, HIVE-22850.2.patch,
> HIVE-22850.3.patch, Screenshot 2020-02-07 at 4.14.51 AM.jpg, jumpTableInfo.png
>
>
> With concurrent queries, time taken for lock acquisition increases
> substantially. As part of lock acquisition in the query,
> {{TxnHandler::checkLock}} gets invoked. This involves getting a mutex and
> compare the locks being requested for, with that of existing locks in
> {{HIVE_LOCKS}} table.
> With concurrent queries, time taken to do this check increase and this
> significantly increases the time taken for getting mutex for other threads
> (due to select for update). In a synthetic workload, it was in the order of
> 10+ seconds. This codepath can be optimized when all lock requests are
> SHARED_READ.
>
>
> !Screenshot 2020-02-07 at 4.14.51 AM.jpg|width=743,height=348!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22850) Optimise lock acquisition in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035177#comment-17035177
]
Zoltan Chovan commented on HIVE-22850:
--
[~rajesh.balamohan], thanks for the clarification! I agree hitting 2k DBs is
unlikely. Fixing it in a separate ticket should be fine.
Also CCing [~dkuzmenko]
> Optimise lock acquisition in TxnHandler
> ---
>
> Key: HIVE-22850
> URL: https://issues.apache.org/jira/browse/HIVE-22850
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-22850.1.patch, HIVE-22850.2.patch,
> HIVE-22850.3.patch, Screenshot 2020-02-07 at 4.14.51 AM.jpg, jumpTableInfo.png
>
>
> With concurrent queries, time taken for lock acquisition increases
> substantially. As part of lock acquisition in the query,
> {{TxnHandler::checkLock}} gets invoked. This involves getting a mutex and
> compare the locks being requested for, with that of existing locks in
> {{HIVE_LOCKS}} table.
> With concurrent queries, time taken to do this check increase and this
> significantly increases the time taken for getting mutex for other threads
> (due to select for update). In a synthetic workload, it was in the order of
> 10+ seconds. This codepath can be optimized when all lock requests are
> SHARED_READ.
>
>
> !Screenshot 2020-02-07 at 4.14.51 AM.jpg|width=743,height=348!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22850) Optimise lock acquisition in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034572#comment-17034572
]
Zoltan Chovan commented on HIVE-22850:
--
[~rajesh.balamohan] I think Peter was referring to this part:
{code:java}
StringBuilder query = new StringBuilder("select count(*) from "
+ "\"HIVE_LOCKS\" where \"HL_DB\" in (");
boolean first = true;
for (String s : dbs) {
if (first) first = false;
else query.append(", ");
query.append('\'');
query.append(s);
query.append('\'');
}
{code}
Oracle db has a limitation on how many elements can be present in and IN()
clause, if it's over 1000 the query will fail. In order to avoid that you might
consider some kind of batching for this part.
> Optimise lock acquisition in TxnHandler
> ---
>
> Key: HIVE-22850
> URL: https://issues.apache.org/jira/browse/HIVE-22850
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-22850.1.patch, HIVE-22850.2.patch,
> HIVE-22850.3.patch, Screenshot 2020-02-07 at 4.14.51 AM.jpg, jumpTableInfo.png
>
>
> With concurrent queries, time taken for lock acquisition increases
> substantially. As part of lock acquisition in the query,
> {{TxnHandler::checkLock}} gets invoked. This involves getting a mutex and
> compare the locks being requested for, with that of existing locks in
> {{HIVE_LOCKS}} table.
> With concurrent queries, time taken to do this check increase and this
> significantly increases the time taken for getting mutex for other threads
> (due to select for update). In a synthetic workload, it was in the order of
> 10+ seconds. This codepath can be optimized when all lock requests are
> SHARED_READ.
>
>
> !Screenshot 2020-02-07 at 4.14.51 AM.jpg|width=743,height=348!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Status: Patch Available (was: Open) > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Attachment: HIVE-22869.patch > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-22869: > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22765) Quote table names in CompactionTxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22765:
-
Attachment: HIVE-22765.6.patch
> Quote table names in CompactionTxnHandler
> -
>
> Key: HIVE-22765
> URL: https://issues.apache.org/jira/browse/HIVE-22765
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: László Pintér
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22765.2.patch, HIVE-22765.3.patch,
> HIVE-22765.4.patch, HIVE-22765.5.patch, HIVE-22765.6.patch, HIVE-22765.patch
>
>
> When I run a metastore backed up by a postgres db, I get the following
> exception:
> {code:java}
> MetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not existMetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not exist Position: 83 at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2468)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2211)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:309)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:311) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:297) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:274) at
> org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:225) at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:157)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.find_next_compact(HiveMetaStore.java:8061)
> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.find_next_compact(Unknown Source) at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.findNextCompact(HiveMetaStoreClient.java:4046)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
> at com.sun.proxy.$Proxy29.findNextCompact(Unknown Source) at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:102){code}
> This is due to missing quotes around table names.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22765) Quote table names in CompactionTxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22765:
-
Attachment: HIVE-22765.5.patch
> Quote table names in CompactionTxnHandler
> -
>
> Key: HIVE-22765
> URL: https://issues.apache.org/jira/browse/HIVE-22765
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: László Pintér
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22765.2.patch, HIVE-22765.3.patch,
> HIVE-22765.4.patch, HIVE-22765.5.patch, HIVE-22765.patch
>
>
> When I run a metastore backed up by a postgres db, I get the following
> exception:
> {code:java}
> MetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not existMetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not exist Position: 83 at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2468)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2211)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:309)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:311) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:297) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:274) at
> org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:225) at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:157)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.find_next_compact(HiveMetaStore.java:8061)
> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.find_next_compact(Unknown Source) at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.findNextCompact(HiveMetaStoreClient.java:4046)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
> at com.sun.proxy.$Proxy29.findNextCompact(Unknown Source) at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:102){code}
> This is due to missing quotes around table names.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22858) HMS broken with mysql
[
https://issues.apache.org/jira/browse/HIVE-22858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033456#comment-17033456
]
Zoltan Chovan commented on HIVE-22858:
--
Hi [~rajesh.balamohan] this was fixed by:
https://issues.apache.org/jira/browse/HIVE-22804
> HMS broken with mysql
> -
>
> Key: HIVE-22858
> URL: https://issues.apache.org/jira/browse/HIVE-22858
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Major
>
> Using commit 7bb1d1edfcba558958265ec47245bc529eaee2d8 (Jan 27) apache master.
> Encountered the following exception when creating new database in hive (mysql
> for HMS).
> https://issues.apache.org/jira/browse/HIVE-22663 may be related to this.
>
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from
> transaction database
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error
> in your SQL syntax; check the manual that corresponds to your MySQL server
> version for the right syntax to use near '"NEXT_TXN_ID" for update' at line 1
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at com.mysql.jdbc.Util.getInstance(Util.java:386)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1054)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4237)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4169)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
> at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:599)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:555)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:7956)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.open_txns(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:19779)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:19764)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:565)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:7956)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(N
[jira] [Updated] (HIVE-22765) Quote table names in CompactionTxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22765:
-
Attachment: HIVE-22765.4.patch
> Quote table names in CompactionTxnHandler
> -
>
> Key: HIVE-22765
> URL: https://issues.apache.org/jira/browse/HIVE-22765
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22765.2.patch, HIVE-22765.3.patch,
> HIVE-22765.4.patch, HIVE-22765.patch
>
>
> When I run a metastore backed up by a postgres db, I get the following
> exception:
> {code:java}
> MetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not existMetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not exist Position: 83 at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2468)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2211)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:309)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:311) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:297) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:274) at
> org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:225) at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:157)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.find_next_compact(HiveMetaStore.java:8061)
> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.find_next_compact(Unknown Source) at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.findNextCompact(HiveMetaStoreClient.java:4046)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
> at com.sun.proxy.$Proxy29.findNextCompact(Unknown Source) at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:102){code}
> This is due to missing quotes around table names.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22850) Optimise lock acquisition in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17032197#comment-17032197
]
Zoltan Chovan commented on HIVE-22850:
--
Could you surround the identifiers in the sql query with \" ? Otherwise it'll
possibly break in postgres/mysql. Also, please test this with the various
backend dbs manually or with docker containers
([https://github.com/apache/hive/blob/master/standalone-metastore/DEV-README])
Additionally, what did you use to benchmark the lock creation?
> Optimise lock acquisition in TxnHandler
> ---
>
> Key: HIVE-22850
> URL: https://issues.apache.org/jira/browse/HIVE-22850
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-22850.1.patch, Screenshot 2020-02-07 at 4.14.51
> AM.jpg
>
>
> With concurrent queries, time taken for lock acquisition increases
> substantially. As part of lock acquisition in the query,
> {{TxnHandler::checkLock}} gets invoked. This involves getting a mutex and
> compare the locks being requested for, with that of existing locks in
> {{HIVE_LOCKS}} table.
> With concurrent queries, time taken to do this check increase and this
> significantly increases the time taken for getting mutex for other threads
> (due to select for update). In a synthetic workload, it was in the order of
> 10+ seconds. This codepath can be optimized when all lock requests are
> SHARED_READ.
>
>
> !Screenshot 2020-02-07 at 4.14.51 AM.jpg|width=743,height=348!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22804) Ensure ANSI quotes are used for mysql connections
[
https://issues.apache.org/jira/browse/HIVE-22804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17027494#comment-17027494
]
Zoltan Chovan commented on HIVE-22804:
--
Note: due to HIVE-22790 currently it's not possible to validate this with
dockerized qtests
> Ensure ANSI quotes are used for mysql connections
> -
>
> Key: HIVE-22804
> URL: https://issues.apache.org/jira/browse/HIVE-22804
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22804.patch
>
>
> Recent changes in direct sql queries to resolve postgres issues(e.g.
> TxnHandler in HIVE-22663 ) break compatibility with mysql backend db.
> A workaround for these issues is to add a session config to the mysql
> connection string, e.g.:
> {code:java}
> jdbc:mysql://localhost:3306/db?sessionVariables=sql_mode=ANSI_QUOTES
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22804) Ensure ANSI quotes are used for mysql connections
[
https://issues.apache.org/jira/browse/HIVE-22804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22804:
-
Status: Patch Available (was: In Progress)
> Ensure ANSI quotes are used for mysql connections
> -
>
> Key: HIVE-22804
> URL: https://issues.apache.org/jira/browse/HIVE-22804
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22804.patch
>
>
> Recent changes in direct sql queries to resolve postgres issues(e.g.
> TxnHandler in HIVE-22663 ) break compatibility with mysql backend db.
> A workaround for these issues is to add a session config to the mysql
> connection string, e.g.:
> {code:java}
> jdbc:mysql://localhost:3306/db?sessionVariables=sql_mode=ANSI_QUOTES
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22804) Ensure ANSI quotes are used for mysql connections
[
https://issues.apache.org/jira/browse/HIVE-22804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22804:
-
Attachment: HIVE-22804.patch
> Ensure ANSI quotes are used for mysql connections
> -
>
> Key: HIVE-22804
> URL: https://issues.apache.org/jira/browse/HIVE-22804
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22804.patch
>
>
> Recent changes in direct sql queries to resolve postgres issues(e.g.
> TxnHandler in HIVE-22663 ) break compatibility with mysql backend db.
> A workaround for these issues is to add a session config to the mysql
> connection string, e.g.:
> {code:java}
> jdbc:mysql://localhost:3306/db?sessionVariables=sql_mode=ANSI_QUOTES
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22804) Ensure ANSI quotes are used for mysql connections
[
https://issues.apache.org/jira/browse/HIVE-22804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan reassigned HIVE-22804:
> Ensure ANSI quotes are used for mysql connections
> -
>
> Key: HIVE-22804
> URL: https://issues.apache.org/jira/browse/HIVE-22804
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
>
> Recent changes in direct sql queries to resolve postgres issues(e.g.
> TxnHandler in HIVE-22663 ) break compatibility with mysql backend db.
> A workaround for these issues is to add a session config to the mysql
> connection string, e.g.:
> {code:java}
> jdbc:mysql://localhost:3306/db?sessionVariables=sql_mode=ANSI_QUOTES
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-22804) Ensure ANSI quotes are used for mysql connections
[
https://issues.apache.org/jira/browse/HIVE-22804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-22804 started by Zoltan Chovan.
> Ensure ANSI quotes are used for mysql connections
> -
>
> Key: HIVE-22804
> URL: https://issues.apache.org/jira/browse/HIVE-22804
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
>
> Recent changes in direct sql queries to resolve postgres issues(e.g.
> TxnHandler in HIVE-22663 ) break compatibility with mysql backend db.
> A workaround for these issues is to add a session config to the mysql
> connection string, e.g.:
> {code:java}
> jdbc:mysql://localhost:3306/db?sessionVariables=sql_mode=ANSI_QUOTES
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22663) Quote all table and column names or do not quote any
[
https://issues.apache.org/jira/browse/HIVE-22663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17026581#comment-17026581
]
Zoltan Chovan commented on HIVE-22663:
--
[~kgyrtkirk] quick workaround is to turn on ansi quotes for the mysql
connection e.g.:
{code:java}
jdbc:mysql://localhost:3306/db?sessionVariables=sql_mode=ANSI_QUOTES
{code}
but a fix is on it's way, although it's a bit difficult to test properly
> Quote all table and column names or do not quote any
>
>
> Key: HIVE-22663
> URL: https://issues.apache.org/jira/browse/HIVE-22663
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Zoltan Chovan
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22663.2.patch, HIVE-22663.3.patch,
> HIVE-22663.4.patch, HIVE-22663.5.patch, HIVE-22663.6.patch, HIVE-22663.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The change in HIVE-22546 is causing following stack trace when I run Hive
> with PostgreSQL as backend db for the metastore.
> 0: jdbc:hive2://localhost:1> create database dumpdb with
> ('repl.source.for'='1,2,3');0: jdbc:hive2://localhost:1> create database
> dumpdb with ('repl.source.for'='1,2,3');Error: Error while compiling
> statement: FAILED: ParseException line 1:28 missing KW_DBPROPERTIES at '('
> near '' (state=42000,code=4)0: jdbc:hive2://localhost:1> create
> database dumpdb with dbproperties ('repl.source.for'='1,2,3');ERROR : FAILED:
> Hive Internal Error: org.apache.hadoop.hive.ql.lockmgr.LockException(Error
> communicating with the
> metastore)org.apache.hadoop.hive.ql.lockmgr.LockException: Error
> communicating with the metastore at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.commitTxn(DbTxnManager.java:541)
> at
> org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:687)
> at
> org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:653)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:969)
> ... stack trace clipped
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)Caused by:
> MetaException(message:Unable to update transaction database
> org.postgresql.util.PSQLException: ERROR: relation
> "materialization_rebuild_locks" does not exist Position: 13 at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at
> This happens because the table names in all the queries in TxnHandler.java
> (including the one at 1312, which causes this stack trace) are not quoting
> the table names. All the tablenames and column names should be quoted there.
> Just the change in HIVE-22546 won't suffice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22765) Quote table names in CompactionTxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024956#comment-17024956
]
Zoltan Chovan commented on HIVE-22765:
--
[~pvary] [~lpinter] Could you review?
> Quote table names in CompactionTxnHandler
> -
>
> Key: HIVE-22765
> URL: https://issues.apache.org/jira/browse/HIVE-22765
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22765.2.patch, HIVE-22765.3.patch, HIVE-22765.patch
>
>
> When I run a metastore backed up by a postgres db, I get the following
> exception:
> {code:java}
> MetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not existMetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not exist Position: 83 at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2468)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2211)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:309)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:311) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:297) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:274) at
> org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:225) at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:157)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.find_next_compact(HiveMetaStore.java:8061)
> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.find_next_compact(Unknown Source) at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.findNextCompact(HiveMetaStoreClient.java:4046)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
> at com.sun.proxy.$Proxy29.findNextCompact(Unknown Source) at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:102){code}
> This is due to missing quotes around table names.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024486#comment-17024486 ] Zoltan Chovan commented on HIVE-22753: -- [~rajesh.balamohan] the code comments state that the elements will have 5s ttl, but this line shows only 1s: .expireAfterWrite(1, TimeUnit.SECONDS) Also do you think having a config value for this would be overkill? Would more time be needed potentially, in case there's a heavy IO load for some reason on the host of HS2? > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, HIVE-22753.2.patch, > HIVE-22753.3.patch, image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22765) Quote table names in CompactionTxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22765:
-
Attachment: HIVE-22765.3.patch
> Quote table names in CompactionTxnHandler
> -
>
> Key: HIVE-22765
> URL: https://issues.apache.org/jira/browse/HIVE-22765
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22765.2.patch, HIVE-22765.3.patch, HIVE-22765.patch
>
>
> When I run a metastore backed up by a postgres db, I get the following
> exception:
> {code:java}
> MetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not existMetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not exist Position: 83 at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2468)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2211)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:309)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:311) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:297) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:274) at
> org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:225) at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:157)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.find_next_compact(HiveMetaStore.java:8061)
> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.find_next_compact(Unknown Source) at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.findNextCompact(HiveMetaStoreClient.java:4046)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
> at com.sun.proxy.$Proxy29.findNextCompact(Unknown Source) at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:102){code}
> This is due to missing quotes around table names.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
