Gabriel39 commented on code in PR #69545:
URL: https://github.com/apache/doris/pull/69545#discussion_r4359933008
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanParams.java:
##########
@@ -204,13 +206,60 @@ public static Table applyOptions(Table table, Map<String,
String> options) {
}
// Apply the relation override first, then normalize each physical
planning branch. A
// single combined copy would flatten independent fallback values into
one sibling's cap.
- Table effectiveTable =
PaimonReaderOptions.runtimeSafeTable(table.copy(isolatedOptions));
+ Table effectiveTable = PaimonReaderOptions.runtimeSafeTable(
+ copyWithHistoricalSchema(table, isolatedOptions, options));
// Validate after every copy so relation options participate in the
documented
// relation > catalog > physical precedence before the effective value
is judged.
PaimonReaderOptions.validateEffectiveTable(effectiveTable);
return effectiveTable;
}
+ static Table copyWithHistoricalSchema(Table table, Map<String, String>
options) {
+ return copyWithHistoricalSchema(table, userOptions(options), options);
+ }
+
+ private static Table copyWithHistoricalSchema(
+ Table table, Map<String, String> dynamicOptions, Map<String,
String> provenance) {
+ if (!(table instanceof FileStoreTable) || table instanceof
ChainGroupReadTable
+ || PaimonTableDecorators.isChainTable(table)) {
+ // Chain groups have their own option filtering and are not
ordinary fallback readers.
+ return table.copy(dynamicOptions);
+ }
+ if (table instanceof FallbackReadFileStoreTable) {
+ FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable)
table;
+ // Each branch has independent snapshot IDs. Keep the SDK's
timestamp translation
+ // before restoring either schema; copying one historical schema
onto the pair is invalid.
+ return PaimonTableDecorators.rebuildFallbackPair(pair,
+ (FileStoreTable) copyWithHistoricalSchema(pair.wrapped(),
dynamicOptions, provenance),
+ (FileStoreTable) copyWithHistoricalSchema(pair.other(),
+ PaimonTableDecorators.otherOptions(pair,
dynamicOptions), provenance));
+ }
+ FileStoreTable dataTable = (FileStoreTable) table;
+ Map<String, String> effectiveOptions = new HashMap<>(table.options());
+ dynamicOptions.forEach((key, value) -> {
+ if (value == null) {
+ effectiveOptions.remove(key);
+ } else {
+ effectiveOptions.put(key, value);
+ }
+ });
+ Optional<Snapshot> snapshot;
+ try {
+ snapshot =
TimeTravelUtil.tryTravelToSnapshot(Options.fromMap(effectiveOptions),
+ dataTable.snapshotManager(), dataTable.tagManager());
+ } catch (Exception e) {
+ // Match the SDK's schema-selection fallback (for example a
timestamp on an empty table).
+ // Its normal copy below still owns option validation and the
eventual scan error.
+ return table.copy(dynamicOptions);
+ }
+ if (snapshot.isPresent() && snapshot.get().schemaId() !=
dataTable.schema().id()) {
+ // Paimon overlays current options onto historical columns.
Restore the matching
+ // sequence/aggregation options first, or renamed fields fail
before scan planning.
+ dataTable = restoreBoundSchema(dataTable,
snapshot.get().schemaId(), provenance, "");
Review Comment:
Fixed in 61e929f7639. Historical schema restoration now preserves the
resolved table's effective startup/read-state keys, including their absence,
before applying explicit catalog and relation overrides. Removing scan.mode
therefore no longer resurrects latest-full when selecting an old snapshot.
Added a real-table unit regression and a cluster regression under both
force_jni_scanner settings. The original code failed with the reported
startup-validation error; the updated cases pass. Full connector install: 866
tests, 0 failures/errors, 1 existing skipped; Checkstyle and all 26 cluster
query checkpoints pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
