[
https://issues.apache.org/jira/browse/HIVE-29512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18073334#comment-18073334
]
Venugopal Reddy K edited comment on HIVE-29512 at 4/14/26 3:31 AM:
-------------------------------------------------------------------
COMPACTION_QUEUE and COMPLETED_COMPACTIONS HMS tables are used for both HIVE
acid table and iceberg table compaction flows. But these backend table records
are cleaned only upon hive acid table/partition drop.
It is straightforward to cleanup them upon drop iceberg table as well at -
[https://github.com/VenuReddy2103/hive/blob/c9e7f5dd6191636232921279acc1a5dd5a6fcaff/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/AcidEventListener.java#L78]
Whereas it is different for partition case. Partition related HMS APIs(like add
partitions, drop partitions etc) are not invoked by HS2 for Iceberg tables.
Hence, thinking of adding a HMS API to cleanup these backend table records for
partition. API can be invoked by clients upon drop partitions of iceberg
tables, leaving the responsibility to clients. Request for community's
opinion/suggestions about it.
was (Author: venureddy):
COMPACTION_QUEUE and COMPLETED_COMPACTIONS HMS tables are used for both HIVE
acid table and iceberg table compaction flows. But these backend table records
are cleaned only upon hive acid table/partition drop.
It is straightforward to cleanup them upon drop iceberg table as well at -
[https://github.com/VenuReddy2103/hive/blob/c9e7f5dd6191636232921279acc1a5dd5a6fcaff/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/AcidEventListener.java#L78]
Whereas it is different for partition case. Partition related HMS APIs(like add
partitions, drop partitions etc) are not invoked by HS2 for Iceberg tables.
Hence, thinking of adding a HMS API to cleanup these backend table records for
partition. API can be invoked by clients upon drop partitions of iceberg
tables, leaving the responsibilty to clients. Request for community's
opinion/suggestions about it.
> COMPACTION_QUEUE and COMPLETED_COMPACTIONS table entries are not cleared upon
> iceberg table delete
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-29512
> URL: https://issues.apache.org/jira/browse/HIVE-29512
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Reporter: Venugopal Reddy K
> Assignee: Venugopal Reddy K
> Priority: Major
>
> *[Description]*
> COMPACTION_QUEUE and COMPLETED_COMPACTIONS table entries are not cleared when
> an iceberg table is deleted. Hence such entries can keep on accumulating over
> time.
> Note: For ACID tables, entries are cleared.
> *[Steps to reproduce]*
> {code:java}
> create database mydb;
> use mydb;
> create table icet1(i int) stored by iceberg
> tblproperties('compactor.threshold.target.size'='1000');
> insert into icet1 values(1),(2),(3);
> insert into icet1 values(1),(2),(3);
> # Wait for COMPACTION_QUEUE to have an entry for table.
> #show compactions; => check state as initiated
> drop table icet1;
> #show compactions; => Still entry exists and it is in initiated state.
> {code}
>
> 1. If the table is recreated again(without adding data) before compaction is
> triggered, compaction fails with below exception. Error message can be seen
> in show compactions output.
> *java.lang.NullPointerException: Cannot invoke
> "org.apache.iceberg.Snapshot.snapshotId()" because the return value of
> "org.apache.iceberg.Table.currentSnapshot()" is null*
> 2. If the table is not created before compaction is triggered, compaction
> fails with error message. Error message can be seen in show compactions
> output -
> *NoSuchObjectException(message:hive.mydb.icet1 table not found)*
> Note: In both the cases, COMPACTION_QUEUE entry for the table is moved to
> COMPLETED_COMPACTIONS upon compaction failure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
