Gabriel39 commented on code in PR #67904:
URL: https://github.com/apache/doris/pull/67904#discussion_r4001749615
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java:
##########
@@ -577,7 +587,26 @@ public Optional<ConnectorMvccSnapshot> beginQuerySnapshot(
Identifier identifier =
Identifier.create(paimonHandle.getDatabaseName(), paimonHandle.getTableName());
long id = latestSnapshotCache.getOrLoad(identifier,
() ->
catalogOps.latestSnapshotId(resolveTable(paimonHandle)).orElse(-1L));
- return
Optional.of(ConnectorMvccSnapshot.builder().snapshotId(id).build());
+ return Optional.of(ConnectorMvccSnapshot.builder().snapshotId(id)
+ .schemaId(statementSchemaId(paimonHandle,
resolveTable(paimonHandle))).build());
Review Comment:
Fixed in `54a5294b6c`. Latest schema capture now retains the actual
statement-scoped schema value, keyed by the branch-aware table handle. Both
schema materialization and column-handle resolution use that value instead of
the catalog-lifetime historical memo when resolving the captured schema ID.
The regression explicitly warms the old historical memo, externally
drops/recreates the table with schema ID 0, and checks the replacement column
names in metadata, handles, and the scan table. A second case also reuses the
data snapshot ID and reads the replacement row through the real Paimon SDK.
Both empty and populated tables pass.
This addresses the latest-schema path introduced by this PR; the
pre-existing historical memo remains in place for historical lookups.
Validation: 1,008 related tests passed, one connectivity test skipped; FE
build, connector packages, and Checkstyle passed.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java:
##########
@@ -483,33 +484,39 @@ public ConnectorTableSchema getTableSchema(
return getTableSchema(session, handle);
}
Table table = loadTable(session, iceHandle);
- Schema schema;
- if (table.currentSnapshot() == null) {
- // Empty table: legacy getSchema falls back to the latest schema
(NEWEST_SCHEMA_ID path).
- schema = table.schema();
- } else {
- schema = table.schemas().get((int) snapshot.getSchemaId());
- if (schema == null) {
- // Defensive: a pinned id absent from table.schemas() (legacy
would NPE) -> latest.
- // INVARIANT: this SLOT-schema fallback MUST stay identical to
the DICT-schema fallback in
- // IcebergScanPlanProvider.pinnedSchema (same getSchemaId()
lookup + same silent -> table.schema()).
- // If the two diverge, the field-id dict names and the BE
scan-slot names resolve DIFFERENT
- // schemas -> BE children.at() std::out_of_range-SIGABRT on a
schema-evolved time-travel read
- // (reverify #65185 L16). Do not harden ONE side to throw
without the other.
- schema = table.schema();
- }
+ Schema schema = resolvePinnedSchema(table, snapshot);
+ String specId =
snapshot.getProperties().get(PARTITION_SPEC_ID_PROPERTY);
+ PartitionSpec spec = specId == null ? table.spec() :
table.specs().get(Integer.parseInt(specId));
Review Comment:
Fixed in `54a5294b6c`. Cached latest pins now carry the table UUID, and
schema/column-handle resolution checks that identity before consuming numeric
schema/spec IDs. UUID-less legacy metadata uses the exact metadata-file
identity. A mismatch invalidates the latest-pin cache and fails the stale
statement with a retry message; it does not substitute only the schema or spec
while retaining an old data fence.
Tests cover missing spec IDs, reused spec IDs, reused schema and spec IDs
together, and changed UUID-less metadata. The replacement table advances
through spec 1 to spec 2, so the reused-ID case specifically exercises a
non-null but wrong historical spec. Retrying resolves the replacement
coordinates without REFRESH TABLE. Existing same-table evolution and cache-hit
load-count tests still pass; the identity check adds no remote table load.
Validation: 1,008 related tests passed, one connectivity test skipped; FE
build, connector packages, and Checkstyle passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
