Gabriel39 commented on code in PR #67904:
URL: https://github.com/apache/doris/pull/67904#discussion_r4011131402
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java:
##########
@@ -100,6 +101,10 @@ public class PaimonConnectorMetadata implements
ConnectorMetadata {
// existing direct-construction tests compile unchanged; production goes
through the 5-arg ctor.
private final PaimonLatestSnapshotCache latestSnapshotCache;
+ // Metadata is statement-scoped: aliases sharing a data fence must also
share one schema generation.
Review Comment:
Fixed in `0481708aea`. Latest Paimon pins now carry schema-file identity
through the retained MVCC snapshot. A replacement metadata scope re-reads that
exact schema once without consulting the historical memo.
`latestPinSurvivesInsertScopeReset` creates and warms the old-generation memo,
recreates the table with a reused schema ID, and verifies the replacement
metadata scope returns the new columns using the original source pin. This
exercises the connector-scope reset boundary used by INSERT replanning.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanParams.java:
##########
@@ -196,14 +201,40 @@ public static FileStoreTable
applyOptionsWithoutTimeTravel(
.filter(key -> !tableOptions.containsKey(key))
.forEach(key -> isolatedOptions.put(key, null));
}
- // The statement fence already selected the schema generation.
Preserve that generation
- // while carrying only the resolved read selector and execution
options into this copy.
+ String schemaId = options.get(BOUND_SCHEMA_ID);
+ if (schemaId != null && table.schema().id() !=
Long.parseLong(schemaId)) {
+ table = restoreBoundSchema(table, Long.parseLong(schemaId));
+ }
FileStoreTable effectiveTable = (FileStoreTable)
PaimonReaderOptions.runtimeSafeTable(
table.copyWithoutTimeTravel(isolatedOptions));
PaimonReaderOptions.validateEffectiveTable(effectiveTable);
return effectiveTable;
}
+ private static FileStoreTable restoreBoundSchema(FileStoreTable table,
long schemaId) {
+ if (table instanceof FallbackReadFileStoreTable) {
+ FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable)
table;
+ // Schema IDs are branch-local. Broadcasting the main schema
through copy(TableSchema)
+ // overwrites the fallback's provenance and can even reset its
branch to main.
+ return new
FallbackReadFileStoreTable(restoreBoundSchema(pair.wrapped(), schemaId),
pair.fallback());
Review Comment:
Fixed in `0481708aea`. The pin captures a separate schema coordinate for
every fallback child, and restoration uses each branch's own coordinate even if
the main schema ID has not changed. The real Paimon regression warms both
branches, advances them to compatible schemas with different IDs, advances the
fallback again after pinning, and verifies the captured row types, branch
identity, and readable rows. A missing fallback child is also rejected.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanParams.java:
##########
@@ -196,14 +201,40 @@ public static FileStoreTable
applyOptionsWithoutTimeTravel(
.filter(key -> !tableOptions.containsKey(key))
.forEach(key -> isolatedOptions.put(key, null));
}
- // The statement fence already selected the schema generation.
Preserve that generation
- // while carrying only the resolved read selector and execution
options into this copy.
+ String schemaId = options.get(BOUND_SCHEMA_ID);
+ if (schemaId != null && table.schema().id() !=
Long.parseLong(schemaId)) {
+ table = restoreBoundSchema(table, Long.parseLong(schemaId));
+ }
FileStoreTable effectiveTable = (FileStoreTable)
PaimonReaderOptions.runtimeSafeTable(
table.copyWithoutTimeTravel(isolatedOptions));
PaimonReaderOptions.validateEffectiveTable(effectiveTable);
return effectiveTable;
}
+ private static FileStoreTable restoreBoundSchema(FileStoreTable table,
long schemaId) {
+ if (table instanceof FallbackReadFileStoreTable) {
+ FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable)
table;
+ // Schema IDs are branch-local. Broadcasting the main schema
through copy(TableSchema)
+ // overwrites the fallback's provenance and can even reset its
branch to main.
+ return new
FallbackReadFileStoreTable(restoreBoundSchema(pair.wrapped(), schemaId),
pair.fallback());
+ }
+ if (table instanceof DelegatedFileStoreTable) {
+ FileStoreTable wrapped = ((DelegatedFileStoreTable)
table).wrapped();
+ return PaimonTableDecorators.replaceWrapped(table,
restoreBoundSchema(wrapped, schemaId));
+ }
+ TableSchema bound = table.schemaManager().schema(schemaId);
+ Map<String, String> persisted =
table.schemaManager().schema(table.schema().id()).options();
+ Map<String, String> merged = new HashMap<>(bound.options());
+ // Field-referencing options evolve with the schema (e.g. bucket-key
and sequence.field
+ // on rename). Only replay the catalog/runtime delta, never another
generation's options.
+ table.options().forEach((key, value) -> {
Review Comment:
Fixed in `0481708aea`. Explicit catalog reader options are now carried
separately into schema restoration and replayed after bound physical options;
relation overrides are applied last. The real Paimon regression covers physical
64/catalog 64 followed by physical 128 and 0, and confirms a relation override
of 32 wins. The external regression suite also covers catalog/relation
precedence on both native and JNI paths; its new assertions are awaiting CI
execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
