-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71792/
-----------------------------------------------------------
(Updated Nov. 21, 2019, 5:35 p.m.)
Review request for hive, Laszlo Pinter and Peter Vary.
Bugs: HIVE-21917
https://issues.apache.org/jira/browse/HIVE-21917
Repository: hive-git
Description
-------
The Initiator thread in the metastore repeatedly loops over entries in the
COMPLETED_TXN_COMPONENTS table to determine which partitions / tables might
need to be compacted. However, entries are never removed from this table except
by a completed Compactor run.
In a cluster where most tables / partitions are write-once read-many, this
results in stale entries in this table never being cleaned up. In a small test
cluster, we have observed approximately 45k entries in this table (virtually
equal to the number of partitions in the cluster) while < 100 of these tables
have delta files at all. Since most of the tables will never get enough writes
to trigger a compaction (and in fact have only ever been written to once), the
initiator thread keeps trying to evaluate them on every loop.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 610cf05204
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
b28b57779b
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
8253ccb9c9
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
268038795b
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
e840758c9d
Diff: https://reviews.apache.org/r/71792/diff/2/
Changes: https://reviews.apache.org/r/71792/diff/1-2/
Testing
-------
Unit tests
Thanks,
Denys Kuzmenko
