[
https://issues.apache.org/jira/browse/IMPALA-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-15242:
------------------------------------
Description:
CatalogServiceCatalog.getOrLoadTable() will reload partitions of a
transactional table when there are stale partitions. However, the check and
update operations are not protected by table write lock. A concurrrent
modification, e.g. EventProcessor processing COMMIT_COMPACTION_EVENT, could
modify the partitions map and cause the reload in getOrLoadTable() fails.
Here is how the stale partitions are checked. It's protected by the table read
lock. It then release the lock.
{code:java}
List<HdfsPartition.Builder> partsToBeRefreshed = Collections.emptyList();
...
readLock(tbl, catalogTimeline);
try {
partsToBeRefreshed =
AcidUtils.getPartitionsForRefreshingFileMetadata(this,
(HdfsTable) tbl);
} finally {
tbl.readLock().unlock();
}{code}
[https://github.com/apache/impala/blob/01062ee897703c0b6c54cfedbe104acf363ff410/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2864-L2865]
The table write lock will be acquired in refreshFileMetadata() when refreshing
the partitions:
{code:java}
if (!partsToBeRefreshed.isEmpty()) {
return refreshFileMetadata((HdfsTable) tbl, partsToBeRefreshed,
catalogTimeline);
}{code}
[https://github.com/apache/impala/blob/01062ee897703c0b6c54cfedbe104acf363ff410/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2891]
refreshFileMetadata() could fail if it uses stale partition ids.
getOrLoadTable() then fails, which consequently fails the coordinator request.
{noformat}
E20260804 19:20:44.249446 134413 JniUtil.java:198] Error in Getting partial
catalog object of TABLE:catalogd_meta_provider_test.insert_only. Time spent:
2s019ms
I20260804 19:20:44.250008 134413 jni-util.cc:335]
java.lang.IllegalStateException: Updating a non existing partition instance
id=25
at
com.google.common.base.Preconditions.checkState(Preconditions.java:586)
at
org.apache.impala.catalog.HdfsTable.updatePartition(HdfsTable.java:1060)
at
org.apache.impala.catalog.HdfsTable.updatePartitions(HdfsTable.java:1051)
at
org.apache.impala.catalog.CatalogServiceCatalog.refreshFileMetadata(CatalogServiceCatalog.java:4549)
at
org.apache.impala.catalog.CatalogServiceCatalog.getOrLoadTable(CatalogServiceCatalog.java:2893)
at
org.apache.impala.catalog.CatalogServiceCatalog.doGetPartialCatalogObject(CatalogServiceCatalog.java:4379)
at
org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:4301)
at
org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:4268)
at
org.apache.impala.service.JniCatalog.lambda$getPartialCatalogObject$10(JniCatalog.java:474)
at
org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
at
org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
at
org.apache.impala.service.JniCatalogOp.execAndSerializeSilentStartAndFinish(JniCatalogOp.java:109)
at
org.apache.impala.service.JniCatalog.execAndSerializeSilentStartAndFinish(JniCatalog.java:256)
at
org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:473){noformat}
The issue can be reproduced by running
CatalogdMetaProviderTest.testTableFileMetadataAfterMinorCompaction after
applying the debug patch [^getOrLoadTable-bug-repro.patch] (on commit
01062ee897).
{noformat}
(pushd fe && mvn test
-Dtest=CatalogdMetaProviderTest#testTableFileMetadataAfterMinorCompaction)
...
[ERROR]
org.apache.impala.catalog.local.CatalogdMetaProviderTest.testTableFileMetadataAfterMinorCompaction
Time elapsed: 41.822 s <<< ERROR!
org.apache.thrift.TException: IllegalStateException: Updating a non existing
partition instance
at
org.apache.impala.catalog.local.CatalogdMetaProvider.sendRequest(CatalogdMetaProvider.java:503)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.loadPartitionsFromCatalogd(CatalogdMetaProvider.java:1145)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.loadPartitionsByRefs(CatalogdMetaProvider.java:1063)
at
org.apache.impala.catalog.local.CatalogdMetaProviderTest.loadPartitions(CatalogdMetaProviderTest.java:212)
at
org.apache.impala.catalog.local.CatalogdMetaProviderTest.loadPartitions(CatalogdMetaProviderTest.java:229)
at
org.apache.impala.catalog.local.CatalogdMetaProviderTest.testFileMetadataAfterCompaction(CatalogdMetaProviderTest.java:685)
at
org.apache.impala.catalog.local.CatalogdMetaProviderTest.testTableFileMetadataAfterMinorCompaction(CatalogdMetaProviderTest.java:607){noformat}
was:
CatalogServiceCatalog.getOrLoadTable() will reload partitions of a
transactional table when there are stale partitions. However, the check and
update operations are not protected by table write lock. A concurrrent
modification, e.g. EventProcessor processing COMMIT_COMPACTION_EVENT, could
modify the partitions map and cause the reload in getOrLoadTable() fails.
Here is how the stale partitions are checked. It's protected by the table read
lock. It then release the lock.
{code:java}
List<HdfsPartition.Builder> partsToBeRefreshed = Collections.emptyList();
...
readLock(tbl, catalogTimeline);
try {
partsToBeRefreshed =
AcidUtils.getPartitionsForRefreshingFileMetadata(this,
(HdfsTable) tbl);
} finally {
tbl.readLock().unlock();
}{code}
[https://github.com/apache/impala/blob/01062ee897703c0b6c54cfedbe104acf363ff410/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2864-L2865]
The table write lock will be acquired in refreshFileMetadata() when refreshing
the partitions:
{code:java}
if (!partsToBeRefreshed.isEmpty()) {
return refreshFileMetadata((HdfsTable) tbl, partsToBeRefreshed,
catalogTimeline);
}{code}
[https://github.com/apache/impala/blob/01062ee897703c0b6c54cfedbe104acf363ff410/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2891]
refreshFileMetadata() could fail if it uses stale partition ids.
getOrLoadTable() then fails, which consequently fails the coordinator request.
{noformat}
E20260804 19:20:44.249446 134413 JniUtil.java:198] Error in Getting partial
catalog object of TABLE:catalogd_meta_provider_test.insert_only. Time spent:
2s019ms
I20260804 19:20:44.250008 134413 jni-util.cc:335]
java.lang.IllegalStateException: Updating a non existing partition instance
id=25
at
com.google.common.base.Preconditions.checkState(Preconditions.java:586)
at
org.apache.impala.catalog.HdfsTable.updatePartition(HdfsTable.java:1060)
at
org.apache.impala.catalog.HdfsTable.updatePartitions(HdfsTable.java:1051)
at
org.apache.impala.catalog.CatalogServiceCatalog.refreshFileMetadata(CatalogServiceCatalog.java:4549)
at
org.apache.impala.catalog.CatalogServiceCatalog.getOrLoadTable(CatalogServiceCatalog.java:2893)
at
org.apache.impala.catalog.CatalogServiceCatalog.doGetPartialCatalogObject(CatalogServiceCatalog.java:4379)
at
org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:4301)
at
org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:4268)
at
org.apache.impala.service.JniCatalog.lambda$getPartialCatalogObject$10(JniCatalog.java:474)
at
org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
at
org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
at
org.apache.impala.service.JniCatalogOp.execAndSerializeSilentStartAndFinish(JniCatalogOp.java:109)
at
org.apache.impala.service.JniCatalog.execAndSerializeSilentStartAndFinish(JniCatalog.java:256)
at
org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:473){noformat}
The issue can be reproduced by running
CatalogdMetaProviderTest.testTableFileMetadataAfterMinorCompaction after
applying the debug patch [^getOrLoadTable-bug-repro.patch] (on commit
01062ee897).
{code}
(pushd fe && mvn test
-Dtest=CatalogdMetaProviderTest#testTableFileMetadataAfterMinorCompaction){code}
> getOrLoadTable should hold table write lock in reloading partitions
> -------------------------------------------------------------------
>
> Key: IMPALA-15242
> URL: https://issues.apache.org/jira/browse/IMPALA-15242
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Priority: Major
> Attachments: getOrLoadTable-bug-repro.patch
>
>
> CatalogServiceCatalog.getOrLoadTable() will reload partitions of a
> transactional table when there are stale partitions. However, the check and
> update operations are not protected by table write lock. A concurrrent
> modification, e.g. EventProcessor processing COMMIT_COMPACTION_EVENT, could
> modify the partitions map and cause the reload in getOrLoadTable() fails.
> Here is how the stale partitions are checked. It's protected by the table
> read lock. It then release the lock.
> {code:java}
> List<HdfsPartition.Builder> partsToBeRefreshed = Collections.emptyList();
> ...
> readLock(tbl, catalogTimeline);
> try {
> partsToBeRefreshed =
> AcidUtils.getPartitionsForRefreshingFileMetadata(this,
> (HdfsTable) tbl);
> } finally {
> tbl.readLock().unlock();
> }{code}
> [https://github.com/apache/impala/blob/01062ee897703c0b6c54cfedbe104acf363ff410/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2864-L2865]
> The table write lock will be acquired in refreshFileMetadata() when
> refreshing the partitions:
> {code:java}
> if (!partsToBeRefreshed.isEmpty()) {
> return refreshFileMetadata((HdfsTable) tbl, partsToBeRefreshed,
> catalogTimeline);
> }{code}
> [https://github.com/apache/impala/blob/01062ee897703c0b6c54cfedbe104acf363ff410/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2891]
> refreshFileMetadata() could fail if it uses stale partition ids.
> getOrLoadTable() then fails, which consequently fails the coordinator request.
> {noformat}
> E20260804 19:20:44.249446 134413 JniUtil.java:198] Error in Getting partial
> catalog object of TABLE:catalogd_meta_provider_test.insert_only. Time spent:
> 2s019ms
> I20260804 19:20:44.250008 134413 jni-util.cc:335]
> java.lang.IllegalStateException: Updating a non existing partition instance
> id=25
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:586)
> at
> org.apache.impala.catalog.HdfsTable.updatePartition(HdfsTable.java:1060)
> at
> org.apache.impala.catalog.HdfsTable.updatePartitions(HdfsTable.java:1051)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.refreshFileMetadata(CatalogServiceCatalog.java:4549)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.getOrLoadTable(CatalogServiceCatalog.java:2893)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.doGetPartialCatalogObject(CatalogServiceCatalog.java:4379)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:4301)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:4268)
> at
> org.apache.impala.service.JniCatalog.lambda$getPartialCatalogObject$10(JniCatalog.java:474)
> at
> org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
> at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerializeSilentStartAndFinish(JniCatalogOp.java:109)
> at
> org.apache.impala.service.JniCatalog.execAndSerializeSilentStartAndFinish(JniCatalog.java:256)
> at
> org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:473){noformat}
> The issue can be reproduced by running
> CatalogdMetaProviderTest.testTableFileMetadataAfterMinorCompaction after
> applying the debug patch [^getOrLoadTable-bug-repro.patch] (on commit
> 01062ee897).
> {noformat}
> (pushd fe && mvn test
> -Dtest=CatalogdMetaProviderTest#testTableFileMetadataAfterMinorCompaction)
> ...
> [ERROR]
> org.apache.impala.catalog.local.CatalogdMetaProviderTest.testTableFileMetadataAfterMinorCompaction
> Time elapsed: 41.822 s <<< ERROR!
> org.apache.thrift.TException: IllegalStateException: Updating a non existing
> partition instance
> at
> org.apache.impala.catalog.local.CatalogdMetaProvider.sendRequest(CatalogdMetaProvider.java:503)
> at
> org.apache.impala.catalog.local.CatalogdMetaProvider.loadPartitionsFromCatalogd(CatalogdMetaProvider.java:1145)
> at
> org.apache.impala.catalog.local.CatalogdMetaProvider.loadPartitionsByRefs(CatalogdMetaProvider.java:1063)
> at
> org.apache.impala.catalog.local.CatalogdMetaProviderTest.loadPartitions(CatalogdMetaProviderTest.java:212)
> at
> org.apache.impala.catalog.local.CatalogdMetaProviderTest.loadPartitions(CatalogdMetaProviderTest.java:229)
> at
> org.apache.impala.catalog.local.CatalogdMetaProviderTest.testFileMetadataAfterCompaction(CatalogdMetaProviderTest.java:685)
> at
> org.apache.impala.catalog.local.CatalogdMetaProviderTest.testTableFileMetadataAfterMinorCompaction(CatalogdMetaProviderTest.java:607){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
