github-actions[bot] commented on code in PR #65782:
URL: https://github.com/apache/doris/pull/65782#discussion_r3608356183
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/helper/IcebergWriterHelper.java:
##########
@@ -178,8 +182,27 @@ private static Metrics buildDataFileMetrics(Table table,
FileFormat fileFormat,
}
}
- return new Metrics(commitData.getRowCount(), columnSizes, valueCounts,
- nullValueCounts, null, lowerBounds, upperBounds);
+ MetricsConfig metricsConfig = MetricsConfig.forTable(table);
+ Schema schema = table.schema();
+ // Physical file stats may contain every column, but manifest metrics
must honor the table's metadata policy.
+ return new Metrics(commitData.getRowCount(),
+ filterDisabledMetrics(columnSizes, schema, metricsConfig),
Review Comment:
[P2] Handle disabled maps in Iceberg stats fetching
An effective `none` mode correctly makes both `columnSizes()` and
`nullValueCounts()` absent, but `StatisticsUtil.getIcebergColumnStats` still
unconditionally adds `task.file().columnSizes().get(colId)` and
`task.file().nullValueCounts().get(colId)` at lines 608/610. After an HMS
Iceberg table is written with this property, enabling
`enable_fetch_iceberg_stats` therefore drives the stats-cache load into a null
dereference/fallback for every affected column. Please update that consumer to
recognize missing metrics (return unknown/empty rather than aggregating a
fabricated zero) and cover the write-to-stats path.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/helper/IcebergWriterHelper.java:
##########
@@ -178,8 +182,27 @@ private static Metrics buildDataFileMetrics(Table table,
FileFormat fileFormat,
}
}
- return new Metrics(commitData.getRowCount(), columnSizes, valueCounts,
- nullValueCounts, null, lowerBounds, upperBounds);
+ MetricsConfig metricsConfig = MetricsConfig.forTable(table);
Review Comment:
[P2] Build the metrics policy once per file batch
`convertToWriterResult` invokes this method once for every
`TIcebergCommitData`, so these new lines rebuild the same `MetricsConfig` and
fetch the same schema for every output file. `MetricsConfig.forTable` walks
schema field IDs/inferred defaults, sort columns, and properties;
partition-heavy or heavily rolled writes therefore repeat fixed table work F
times on the synchronous FE commit path. The transaction table's metadata is
stable for this batch, so please capture the config/schema once before the loop
and pass them into `buildDataFileMetrics`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
