andygrove opened a new issue, #5642: URL: https://github.com/apache/datafusion-comet/issues/5642
### What is the problem the feature request solves? Native Iceberg writes fall back when `write.metadata.metrics.default` or a per-column `write.metadata.metrics.column.*` is `counts` or `none` (`CometIcebergNativeWrite` trigger rules, pinned by `CometIcebergWriteDetectionSuite`). The restriction was introduced in #5298 because iceberg-rust could not produce those metrics modes, and `iceberg-writes.md` says it "remains only until that assembly lands". #5361 moved manifest metrics assembly to the JVM: `DataFile` metrics are rebuilt at commit time from the written file's footer using Iceberg's own `MetricsConfig` / `ParquetUtil` code, with NaN counts supplied by the native writer. The parity tests in `CometIcebergWriteActionSuite` already cover mode `none` with a per-column `full` override and mode `counts` with `WRITE ORDERED BY`. That is the machinery the restriction was waiting for, so the restriction may now be liftable with no change to the writer. ### Describe the potential solution 1. Confirm what the JVM rebuild produces for `counts` (counts only, no bounds) and `none` (no metrics) against iceberg-java output for the same data, including the sorted-column promotion that `MetricsConfig` applies under `counts`. 2. If identical, delete the `counts` / `none` trigger rules, update the allowlist table in `iceberg-writes.md`, and turn the existing detection-suite fall-back tests into parity tests. 3. If not identical, document the specific divergence and keep the rule. ### Additional context Part of the native Iceberg writes epic. Related: #5298 (detection), #5361 (native writer and JVM metrics assembly). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
