[
https://issues.apache.org/jira/browse/IMPALA-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18074766#comment-18074766
]
Quanlong Huang commented on IMPALA-14905:
-----------------------------------------
Checked the FeSupport.INFO logs, file metadata is reloaded after the major
compaction. But catalogd still sees 3 files:
{noformat}
I20260419 05:30:49.773257 1590002 HiveJdbcClientPool.java:102] Executing sql :
alter table partial_catalog_info_test.insert_only compact 'major' and wait
I20260419 05:35:57.920764 1590002 HdfsTable.java:799] Loaded file and block
metadata for partial_catalog_info_test.insert_only. Time taken: 1.570ms. Files:
3, Blocks: 3, Total size: 1.23KB, File sizes (min/avg/max): 358B/420B/451B,
Modification times (min/max): 2026-04-19 05:30:45/2026-04-19 05:30:49, Access
times (min/max): 2026-04-19 05:30:45/2026-04-19 05:30:49, Hosts: 3, Host:Disk
pairs: 6{noformat}
In HS2 logs (cluster/hive/hive-server2.log), I can see the major compaction
finished:
{noformat}
2026-04-19T05:35:57,915 INFO
[impala-ec2-redhat86-m6i-4xlarge-ondemand-1545.vpc.cloudera.com-51_executor]
service.AcidCompactionService: Completed MAJOR compaction for
partial_catalog_info_test.insert_only in txnId=2991, lockId=608 (TxnStatus:
'o'), marking as compacted.{noformat}
The problem might be in AcidUtils.getPartitionsForRefreshingFileMetadata() that
the compaction is not detected or compared correctly.
https://github.com/apache/impala/blob/7ea5c35245b49629c6a35b38041ff0144d6dd9a1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2851
https://github.com/apache/impala/blob/7ea5c35245b49629c6a35b38041ff0144d6dd9a1/fe/src/main/java/org/apache/impala/util/AcidUtils.java#L800
> testTableFileMetadataAfterMajorCompaction
> ------------------------------------------
>
> Key: IMPALA-14905
> URL: https://issues.apache.org/jira/browse/IMPALA-14905
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> PartialCatalogInfoWriteIdTest.testTableFileMetadataAfterMajorCompaction is
> flaky that it could fail in wrong file count:
> {noformat}
> java.lang.AssertionError:
> Actual file_descriptors:
> [delta_0000002_0000002_0000/000000_0, delta_0000003_0000003_0000/000000_0,
> delta_1_1/c44c280eb00f05b4-44ee388d00000000_685550147_data.0.parq]
> Actual insert_file_descriptors:
> []
> Actual delete_file_descriptors:
> [] expected:<1> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at
> org.apache.impala.catalog.PartialCatalogInfoWriteIdTest.testFileMetadataAfterCompaction(PartialCatalogInfoWriteIdTest.java:621)
> at
> org.apache.impala.catalog.PartialCatalogInfoWriteIdTest.testTableFileMetadataAfterMajorCompaction(PartialCatalogInfoWriteIdTest.java:516)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> ...{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
