github-actions[bot] commented on code in PR #69983:
URL: https://github.com/apache/doris/pull/69983#discussion_r4442499089
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalMetaCache.java:
##########
@@ -990,6 +999,11 @@ private void
invalidateRelatedEntriesOnTableRemoval(NameMapping nameMapping) {
if (snapshots != null) {
snapshots.invalidateIf(key ->
key.getNameMapping().equals(nameMapping));
}
+ MetaCacheEntry<IcebergSnapshotEntryKey, IcebergPartitionInfo>
partitions =
+ partitionEntry.getIfInitialized(nameMapping.getCtlId());
+ if (partitions != null) {
+ partitions.invalidateIf(key ->
key.getNameMapping().equals(nameMapping));
Review Comment:
[P2] Retire partition entries during table-generation replacement
This cleanup only runs on table removal. Normal asynchronous refresh calls
`retireTableGeneration`, and `REPLACED` deliberately skips the removal
listener, so old partition keys survive even though superseded snapshot
projections are invalidated. Each consumed generation now leaves another full
partition map cached; after its pinned queries finish, new latest readers use
the new metadata-file key and explicit historical reads do not use this entry.
The default strong count cache can retain these obsolete full views for 24
hours (up to 1000 entries), multiplying partition-metadata memory compared with
the previous in-snapshot storage. Extend generation-aware retirement to
partition entries as well, including preventing an old loader from republishing
retired keys, while preserving same-generation reuse.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergSnapshotCacheValue.java:
##########
@@ -137,9 +139,18 @@ private IcebergSnapshotCacheValue(IcebergPartitionInfo
partitionInfo, IcebergSna
}
public IcebergPartitionInfo getPartitionInfo() {
+ return partitionInfoLoader == null ? partitionInfo :
partitionInfoLoader.get();
Review Comment:
[P2] Avoid reloading the entire partition view per MV partition
With `meta.cache.iceberg.partition.enable=false` (or a partition view
rejected by its weight limit), this invokes the full loader on every access.
`MTMVRefreshContext.loadSnapshots` calls the default
`MTMVRelatedTableIf.getPartitionSnapshots`, which loops over N names and
reaches this method through `IcebergExternalTable.getPartitionSnapshot` each
time. One refresh therefore performs N full `PartitionsTable.planFiles()` scans
and map rebuilds for the same pinned snapshot; previously the snapshot retained
one loaded view. Reuse the view across that bulk operation or in the refresh
scope, while preserving shared-cache accounting, and cover repeated accesses
when admission is disabled/rejected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
