924060929 commented on code in PR #69983:
URL: https://github.com/apache/doris/pull/69983#discussion_r4446656036
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalMetaCache.java:
##########
@@ -1167,27 +1189,41 @@ private IcebergSnapshotCacheValue
loadSnapshotProjection(
}
try {
IcebergSnapshot latestIcebergSnapshot =
IcebergUtils.getLatestIcebergSnapshot(projectionTable);
- IcebergPartitionInfo icebergPartitionInfo;
- // The projection and its partition eligibility must be classified
from the same
- // retained table generation. The no-argument method performs
another cache lookup,
- // which can cross a concurrent refresh/reset boundary.
+ // Ordinary scans and SQL-cache versions need only this frozen
metadata generation.
+ // Partition consumers load the separate, generation-keyed and
budgeted view on demand.
+ IcebergPartitionInfo icebergPartitionInfo =
IcebergPartitionInfo.empty();
boolean validRelatedTable =
IcebergUtils.isValidRelatedTable(projectionTable);
- if (!validRelatedTable) {
- icebergPartitionInfo = IcebergPartitionInfo.empty();
- } else {
- icebergPartitionInfo =
IcebergUtils.loadPartitionInfo(dorisTable, projectionTable,
- latestIcebergSnapshot.getSnapshotId(),
latestIcebergSnapshot.getSchemaId(), authenticator,
- enableMappingVarbinary, enableMappingTimestampTz);
- }
Optional<Map<Integer, List<String>>> nameMapping =
IcebergUtils.getNameMapping(projectionTable);
- return isolateForQueries
+ IcebergSnapshotCacheValue value = isolateForQueries
? new IcebergSnapshotCacheValue(
icebergPartitionInfo, latestIcebergSnapshot,
nameMapping,
retainedTable, retainedCurrentSnapshotJson)
: new IcebergSnapshotCacheValue(
icebergPartitionInfo, latestIcebergSnapshot,
nameMapping,
retainedTable);
+ if (validRelatedTable) {
+ NameMapping tableIdentity = dorisTable.getOrBuildNameMapping();
+ String displayName = dorisTable.getCatalog().getName() + "." +
dorisTable.getDbName()
+ + "." + dorisTable.getName();
+ Optional<IcebergSnapshotEntryKey> key =
IcebergSnapshotEntryKey.tryCreate(tableIdentity, retainedTable);
+ MetaCacheEntry<IcebergSnapshotEntryKey, IcebergPartitionInfo>
partitions =
+ key.isPresent() ?
partitionEntry.get(key.get().getNameMapping().getCtlId()) : null;
Review Comment:
Fixed in 1385a1ea0a7a5f97a6aa2755300c1047ef34b044. Lazy partition loading
now derives partition columns from the same retained table/schema under its
captured authenticator and mapping options; it no longer resolves a current
schema-cache/catalog group. Projection creation captures only an existing
shared partition entry, while retired-group and session-catalog paths load
directly. The existing admission and generation-retirement fences are
preserved. Added three tests with real Iceberg day-partition metadata and a
table-generation lease held across reset, covering reset before the first read,
reset before projection creation, and a session-catalog path with no cache
group. All 204 related FE tests and full FE packaging/Checkstyle pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
