andygrove opened a new issue, #5643: URL: https://github.com/apache/datafusion-comet/issues/5643
### What is the problem the feature request solves? The native Iceberg write eligibility gate (`CometIcebergNativeWrite`, documented in `iceberg-writes.md`) is an allowlist, and each restriction it enforces was added conservatively in #5298 so the writer could land incrementally. None of the restrictions has an issue saying whether it is permanent or what would lift it, so there is no way to tell how much of the fallback surface is intended. Current restrictions, each of which makes a write fall back to iceberg-java: - [ ] format version 3 or newer (`MinUnsupportedFormatVersion`) - [ ] any `encryption.*` property - [ ] `write.parquet.bloom-filter-max-bytes` and per-column bloom filter enablement - [ ] `write.parquet.page-version=v2` - [ ] `write.parquet.shred-variants=true` (also depends on #5433 / the Variant epic #5438) - [ ] any unvetted `write.parquet.*` key, and any `parquet.*` table property (including `parquet.enable.dictionary`) - [ ] any `parquet.*` key in the session Hadoop configuration (other than `parquet.hadoop.vectored.io.enabled`) - [ ] non-default `write.parquet.row-group-check-min-record-count` / `max-record-count` - [ ] `write.object-storage.enabled=true` and `write.location-provider.impl` - [ ] `io-impl` set, or a `FileIO` outside the recognized class hierarchy - [ ] data location schemes other than `file`, `memory`, `s3`, `s3a`, `gs`, `oss` (and see #5637 for `gs`) - [ ] metrics mode `counts` / `none` (tracked separately in the metrics-mode issue) ### Describe the potential solution For each restriction, record one of three decisions in this issue and in `iceberg-writes.md`: - **Lift**: the native path already reproduces iceberg-java's outcome, or can with a bounded change; file a follow-up and link it. - **Keep, pending upstream**: needs an iceberg-rust or parquet-rs feature; link the upstream issue. - **Keep, permanent**: intentionally out of scope (for example encryption); say why. The ordering should follow how often the setting appears in real tables. Object-storage layout (`write.object-storage.enabled`) and bloom filters are the two most likely to be hit by production tables and should be decided first. ### Additional context Part of the native Iceberg writes epic. Related: #5298 (introduced the allowlist), #5361 (native writer). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
