924060929 commented on code in PR #67904:
URL: https://github.com/apache/doris/pull/67904#discussion_r4004077344
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java:
##########
@@ -100,6 +101,10 @@ public class PaimonConnectorMetadata implements
ConnectorMetadata {
// existing direct-construction tests compile unchanged; production goes
through the 5-arg ctor.
private final PaimonLatestSnapshotCache latestSnapshotCache;
+ // Metadata is statement-scoped: aliases sharing a data fence must also
share one schema generation.
Review Comment:
The captured schema is owned by one ConnectorMetadata instance, but an
INSERT planning retry resets the connector scope while retaining the source
MVCC snapshot. The replacement metadata has an empty statementSchemas map, so a
preserved latest pin with a schema ID falls through to the catalog-lifetime
schemaAtMemo. With an old table generation still memoized and a recreated
same-name table reusing that schema ID, attempt 1 can bind the new table while
attempt 2 obtains old column handles and fails. Please make the exact captured
schema live as long as the retained MVCC pin, or re-read it once in the
replacement scope without consulting the historical memo. Add an INSERT replan
regression with an old-generation memo.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanParams.java:
##########
@@ -196,14 +201,40 @@ public static FileStoreTable
applyOptionsWithoutTimeTravel(
.filter(key -> !tableOptions.containsKey(key))
.forEach(key -> isolatedOptions.put(key, null));
}
- // The statement fence already selected the schema generation.
Preserve that generation
- // while carrying only the resolved read selector and execution
options into this copy.
+ String schemaId = options.get(BOUND_SCHEMA_ID);
+ if (schemaId != null && table.schema().id() !=
Long.parseLong(schemaId)) {
+ table = restoreBoundSchema(table, Long.parseLong(schemaId));
+ }
FileStoreTable effectiveTable = (FileStoreTable)
PaimonReaderOptions.runtimeSafeTable(
table.copyWithoutTimeTravel(isolatedOptions));
PaimonReaderOptions.validateEffectiveTable(effectiveTable);
return effectiveTable;
}
+ private static FileStoreTable restoreBoundSchema(FileStoreTable table,
long schemaId) {
+ if (table instanceof FallbackReadFileStoreTable) {
+ FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable)
table;
+ // Schema IDs are branch-local. Broadcasting the main schema
through copy(TableSchema)
+ // overwrites the fallback's provenance and can even reset its
branch to main.
+ return new
FallbackReadFileStoreTable(restoreBoundSchema(pair.wrapped(), schemaId),
pair.fallback());
+ }
+ if (table instanceof DelegatedFileStoreTable) {
+ FileStoreTable wrapped = ((DelegatedFileStoreTable)
table).wrapped();
+ return PaimonTableDecorators.replaceWrapped(table,
restoreBoundSchema(wrapped, schemaId));
+ }
+ TableSchema bound = table.schemaManager().schema(schemaId);
+ Map<String, String> persisted =
table.schemaManager().schema(table.schema().id()).options();
+ Map<String, String> merged = new HashMap<>(bound.options());
+ // Field-referencing options evolve with the schema (e.g. bucket-key
and sequence.field
+ // on rename). Only replay the catalog/runtime delta, never another
generation's options.
+ table.options().forEach((key, value) -> {
Review Comment:
Value equality cannot recover option provenance. For example, an explicit
catalog read.batch-size=64 is indistinguishable here from an old physical value
of 64. After an external option-only change makes the physical value 128, this
filter drops the explicit catalog override and the scan incorrectly uses 128; a
new value of 0 can instead make a previously valid query fail validation.
Preserve explicit catalog/runtime overrides separately and merge bound physical
options, then catalog/runtime overrides, then relation overrides.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanParams.java:
##########
@@ -196,14 +201,40 @@ public static FileStoreTable
applyOptionsWithoutTimeTravel(
.filter(key -> !tableOptions.containsKey(key))
.forEach(key -> isolatedOptions.put(key, null));
}
- // The statement fence already selected the schema generation.
Preserve that generation
- // while carrying only the resolved read selector and execution
options into this copy.
+ String schemaId = options.get(BOUND_SCHEMA_ID);
+ if (schemaId != null && table.schema().id() !=
Long.parseLong(schemaId)) {
+ table = restoreBoundSchema(table, Long.parseLong(schemaId));
+ }
FileStoreTable effectiveTable = (FileStoreTable)
PaimonReaderOptions.runtimeSafeTable(
table.copyWithoutTimeTravel(isolatedOptions));
PaimonReaderOptions.validateEffectiveTable(effectiveTable);
return effectiveTable;
}
+ private static FileStoreTable restoreBoundSchema(FileStoreTable table,
long schemaId) {
+ if (table instanceof FallbackReadFileStoreTable) {
+ FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable)
table;
+ // Schema IDs are branch-local. Broadcasting the main schema
through copy(TableSchema)
+ // overwrites the fallback's provenance and can even reset its
branch to main.
+ return new
FallbackReadFileStoreTable(restoreBoundSchema(pair.wrapped(), schemaId),
pair.fallback());
Review Comment:
This restores only the main child and keeps the cached fallback child
unchanged. If a warm M0/F0 fallback table has since received the same
compatible nullable ADD COLUMN on both real branches, the current state is
M1/F1 but Doris constructs M1/F0. Paimon 1.3.1 validates the complete row types
before projection, so even SELECT id fails although the external branches are
compatible. Capture and restore each child using its own branch-local schema
coordinate; a main schema ID cannot be broadcast to the fallback branch.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java:
##########
@@ -172,6 +172,19 @@ private PluginDrivenMvccSnapshot materializeLatest(
// legacy listPartitions/LIST/timestamp path below (byte-unchanged;
the no-op applySnapshot for the
// latest pin is side-effect-free for both paimon and iceberg).
ConnectorTableHandle pinnedHandle = metadata.applySnapshot(session,
handle, connectorSnapshot);
+ PluginDrivenSchemaCacheValue pinnedSchema = null;
+ if (connectorSnapshot.getSchemaId() >= 0) {
+ // Latest data and schema can advance independently. Keep the
connector's exact schema
+ // on the statement pin so analysis cannot fall back to a
different cached generation.
+ ConnectorTableSchema atSchema = metadata.getTableSchema(session,
pinnedHandle, connectorSnapshot);
Review Comment:
The newly retained latest schema calls the snapshot-aware metadata overload.
Through an HMS catalog, Hive delegates that overload to the Iceberg sibling but
does not apply the capability reflection used by the ordinary two-argument
overload. The resulting pinned schema has an empty capability set and shadows
the enriched schema, disabling TopN lazy materialization, nested-column
pruning, and storage predicate inference for ordinary HMS Iceberg reads. Please
reuse the existing sibling capability inheritance helper in the snapshot-aware
Hive path and add a retained-latest publication test for all three capabilities.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
