Gabriel39 commented on code in PR #66247:
URL: https://github.com/apache/doris/pull/66247#discussion_r3698073950
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -523,64 +628,156 @@ static Optional<Long> parseDataSizeBytes(String value) {
if (value == null || value.trim().isEmpty()) {
return Optional.empty();
}
- String normalized = value.trim().toLowerCase(Locale.ROOT).replace("_",
"").replace(" ", "");
- int unitStart = 0;
- while (unitStart < normalized.length()
- && (Character.isDigit(normalized.charAt(unitStart)) ||
normalized.charAt(unitStart) == '.')) {
- unitStart++;
- }
- if (unitStart == 0) {
- return Optional.empty();
- }
try {
- double number = Double.parseDouble(normalized.substring(0,
unitStart));
- String unit = normalized.substring(unitStart);
- long multiplier;
- switch (unit) {
- case "":
- case "b":
- case "byte":
- case "bytes":
- multiplier = 1L;
- break;
- case "k":
- case "kb":
- case "kib":
- multiplier = 1024L;
- break;
- case "m":
- case "mb":
- case "mib":
- multiplier = 1024L * 1024L;
- break;
- case "g":
- case "gb":
- case "gib":
- multiplier = 1024L * 1024L * 1024L;
- break;
- case "t":
- case "tb":
- case "tib":
- multiplier = 1024L * 1024L * 1024L * 1024L;
- break;
- default:
- return Optional.empty();
- }
- return Optional.of((long) (number * multiplier));
- } catch (NumberFormatException e) {
+ // Keep the BE guard's accepted grammar identical to the Paimon
option parser that will
+ // consume this value; accepting a superset lets invalid
serialized options reach scans.
+ return Optional.of(MemorySize.parse(value).getBytes());
+ } catch (IllegalArgumentException e) {
return Optional.empty();
}
}
private void initTable() {
Preconditions.checkState(params.containsKey("serialized_table"));
table = PaimonUtils.deserialize(params.get("serialized_table"));
+ String encodedSystemSource = params.get(PAIMON_OPTION_PREFIX +
DORIS_SERIALIZED_SYSTEM_SOURCE);
+ FileStoreTable systemSource = encodedSystemSource == null
+ ? null : PaimonUtils.deserialize(encodedSystemSource);
+ table = applyBackendManifestParallelism(table,
+ params.get(PAIMON_OPTION_PREFIX +
DORIS_MANIFEST_PARALLELISM_CAP),
+ Runtime.getRuntime().availableProcessors(), systemSource,
+ params.get(PAIMON_OPTION_PREFIX + DORIS_SYSTEM_TABLE_TYPE));
+ validateSerializedReaderOptions(table);
Review Comment:
Fixed in 8e22314788. The legacy-wrapper validator descends through the
private system source and fallback pair. A real serialized audit_log wrapper
with fallback-only read.batch-size=0 now proves the backstop.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java:
##########
@@ -636,6 +637,18 @@ private Optional<LogicalPlan> handleMetaTable(TableIf
table, UnboundRelation unb
if (sysTablePlan.isNative()) {
List<String> qualifierWithoutTableName =
qualifiedTableName.subList(0, qualifiedTableName.size() - 1);
ExternalTable sysExternalTable =
sysTablePlan.getSysExternalTable();
+ if (sysExternalTable instanceof PluginDrivenSysExternalTable) {
+ Optional<TableSnapshot> tableSnapshot =
unboundRelation.getTableSnapshot();
+ Optional<TableScanParams> scanParams =
Optional.ofNullable(unboundRelation.getScanParams());
+ StatementContext statementContext =
cascadesContext.getStatementContext();
+ ((PluginDrivenSysExternalTable)
sysExternalTable).resolveScanPin(
Review Comment:
Fixed in 8e22314788. System-table row-count planning consumes the relation
memo connector snapshot through the snapshot-aware statistics API and preserves
UNKNOWN instead of reopening latest. Positive and empty pins are covered.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -1226,6 +1288,12 @@ private Table resolveSchemaDictTable(Table table,
PaimonTableHandle handle) {
return table;
}
if (table instanceof ReadOptimizedTable) {
+ FileStoreTable pinnedSource = handle.getSysBaseTable();
Review Comment:
Fixed in 8e22314788. The read-optimized schema dictionary now applies the
relation scan options to the captured source, with a historical snapshot
regression across a column rename.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExecuteCommand.java:
##########
@@ -95,22 +96,25 @@ public void run(ConnectContext ctx, StmtExecutor executor)
throws Exception {
// per-statement scope so a prior execution's cached tables/state
never leak into this one (the
// scope key's queryId is a second line of defense). See
StatementContext#resetConnectorStatementScope.
statementContext.resetConnectorStatementScope();
+ statementContext.resetMvccSnapshots();
LogicalPlan logicalPlan = prepareCommand.getLogicalPlan();
- LogicalPlan relationRoot = logicalPlan;
+ List<LogicalPlan> relationRoots = new ArrayList<>();
if (logicalPlan instanceof InsertIntoTableCommand) {
- relationRoot = ((InsertIntoTableCommand)
logicalPlan).getLogicalQuery();
+ relationRoots.add(((InsertIntoTableCommand)
logicalPlan).getLogicalQuery());
} else if (logicalPlan instanceof InsertOverwriteTableCommand) {
- relationRoot = ((InsertOverwriteTableCommand)
logicalPlan).getLogicalQuery();
+ relationRoots.add(((InsertOverwriteTableCommand)
logicalPlan).getLogicalQuery());
} else if (logicalPlan instanceof UpdateCommand) {
- relationRoot = ((UpdateCommand) logicalPlan).getLogicalQuery();
- } else if (logicalPlan instanceof Command) {
- // Non-DML commands deliberately have no traversable children;
they cannot own a
- // relation scan tree whose resolved state needs resetting.
- relationRoot = null;
+ relationRoots.add(((UpdateCommand) logicalPlan).getLogicalQuery());
+ } else if (logicalPlan instanceof DeleteFromUsingCommand) {
Review Comment:
Fixed in 8e22314788 and synchronized to the 4.1 PR. Execute now exposes the
base DELETE root and recursively follows expression-owned subquery plans. The
regression executes the prepared DELETE repeatedly and observes a fresh option
resolution each time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
