Gabriel39 commented on code in PR #67904:
URL: https://github.com/apache/doris/pull/67904#discussion_r3999341615
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java:
##########
@@ -577,7 +584,14 @@ public Optional<ConnectorMvccSnapshot> beginQuerySnapshot(
Identifier identifier =
Identifier.create(paimonHandle.getDatabaseName(), paimonHandle.getTableName());
long id = latestSnapshotCache.getOrLoad(identifier,
() ->
catalogOps.latestSnapshotId(resolveTable(paimonHandle)).orElse(-1L));
- return
Optional.of(ConnectorMvccSnapshot.builder().snapshotId(id).build());
+ return Optional.of(ConnectorMvccSnapshot.builder().snapshotId(id)
+ .schemaId(statementSchemaId(paimonHandle,
resolveTable(paimonHandle))).build());
+ }
+
+ private long statementSchemaId(PaimonTableHandle handle, Table table) {
+ return statementSchemaIds.computeIfAbsent(handle,
+ ignored ->
catalogOps.latestSchema(table).map(PaimonCatalogOps.PaimonSchemaSnapshot::schemaId)
Review Comment:
In scope: this PR introduces statement schema capture, eager pinned-schema
materialization and exact schema restoration. Fixed in 56cfe11da7. Latest/exact
schema reads and scan/backend/system-source restoration now execute inside the
connector authentication/TCCL scope. Three tests guard real schema FileIO for a
warm latest-cache hit, pinned-schema materialization and scan restoration; all
fail before the fix and pass afterward. The full Paimon package build passes
with 576 tests passed and one skipped.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java:
##########
@@ -172,6 +172,14 @@ private PluginDrivenMvccSnapshot materializeLatest(
// legacy listPartitions/LIST/timestamp path below (byte-unchanged;
the no-op applySnapshot for the
// latest pin is side-effect-free for both paimon and iceberg).
ConnectorTableHandle pinnedHandle = metadata.applySnapshot(session,
handle, connectorSnapshot);
+ PluginDrivenSchemaCacheValue pinnedSchema = null;
+ if (connectorSnapshot.getSchemaId() >= 0) {
+ // Latest data and schema can advance independently. Keep the
connector's exact schema
+ // on the statement pin so analysis cannot fall back to a
different cached generation.
+ ConnectorTableSchema atSchema = metadata.getTableSchema(session,
pinnedHandle, connectorSnapshot);
+ pinnedSchema = toSchemaCacheValue(metadata, session,
Review Comment:
In scope: this PR introduces the local pinned schema, so its partition
consumers must use that same schema before StatementContext publishes the pin.
Fixed in 56cfe11da7. The style gate and RANGE/LIST builders receive the local
pinned partition columns; incremental reads continue using their latest schema.
Three tests cover partition type/arity evolution and an ambient unpartitioned
schema. All fail before the fix and pass afterward; all 71 FE MVCC tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
