[
https://issues.apache.org/jira/browse/HIVE-25868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17598018#comment-17598018
]
Rohan Nimmagadda commented on HIVE-25868:
-----------------------------------------
We have more than 2M completed transactions in HIVE, with default HIVE
properties the backend DB (Postgres) is not able to handle the query to delete
in bigger chunks it failed out with the below expectation
{code:java}
An I/O error occurred while sending to the backend. (SQLState=08006,
ErrorCode=0)
2022-08-25T15:06:00,256 ERROR [pool-6-thread-6]:
txn.AcidCompactionHistoryService (AcidCompactionHistoryService.java:run(64)) -
Serious error in pool-6-thread-6
org.apache.hadoop.hive.metastore.api.MetaException: Unable to connect to
transaction database org.postgresql.util.PSQLException: An I/O error occurred
while sending to the backend.
purgeCompactionHistory() : An I/O error occurred while sending to the backend
{code}
So we applied the [HIVE-25659|https://issues.apache.org/jira/browse/HIVE-25659]
to HIVE 3.1 Version and added the below configs to delete older completed txn's
The below configurations should be documented
# hive.direct.sql.max.parameters=10000
# hive.metastore.housekeeping.threads.on=true
# hive.metastore.task.threads.remote=true
# hive.compactor.history.retention.succeeded=1
#
hive.compactor.history.retention.failed=3
> AcidHouseKeeperService fails to purgeCompactionHistory if the entries in
> COMPLETED_COMPACTIONS tables
> ------------------------------------------------------------------------------------------------------
>
> Key: HIVE-25868
> URL: https://issues.apache.org/jira/browse/HIVE-25868
> Project: Hive
> Issue Type: Bug
> Components: Hive, Metastore, Standalone Metastore
> Reporter: mahesh kumar behera
> Assignee: mahesh kumar behera
> Priority: Major
>
> To purge the entries, prepared statement is created. If the number of entries
> in the prepared statement goes beyond the limit of backend db (for postgres
> it around 32k), then the operation fails.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
