andygrove opened a new issue, #6114: URL: https://github.com/apache/datafusion-comet/issues/6114
### What is the problem the feature request solves? For high-cardinality columns, the native Iceberg writer produces Parquet files with a large dictionary page where iceberg-java produces none, so selective reads of native-written files fetch several MB more than they need to. parquet-mr, which iceberg-java writes through, drops dictionary encoding for a column chunk when the dictionary is not paying for itself: the first time it checks, if the dictionary-encoded size plus the dictionary is not smaller than the raw size, it switches the whole column chunk to plain and writes no dictionary page. parquet-rs has no such check. It keeps dictionary encoding until the dictionary reaches `dictionary_page_size_limit` (Iceberg's `write.parquet.dict-size-bytes`, 2 MB by default), then falls back to plain for the rest of the chunk and still writes the full dictionary page. Measured on the fixture of `CometIcebergNativeSuite` "native scan skips pages within a single row group": 1M rows of `id BIGINT` (sorted, unique) and `payload STRING` (16 hex characters, unique), one 512 MB row group, uncompressed, 2000-row pages. | | iceberg-java (parquet-mr 1.17.1) | native (parquet-rs) | |---|---|---| | `id` encodings | `PLAIN, BIT_PACKED`, no dictionary page | `RLE_DICTIONARY, PLAIN, RLE`, 2.10 MB dictionary page | | `payload` encodings | `PLAIN, RLE, BIT_PACKED`, no dictionary page | `RLE_DICTIONARY, PLAIN, RLE`, 2.10 MB dictionary page | | bytes scanned for `WHERE id BETWEEN 1000 AND 1100` | 0.6 MB | 4.7 MB (of a 20.7 MB file) | Every read that touches a column chunk has to fetch its dictionary page, so page-index pruning can't bring a selective read below roughly 2 MB per such column per row group. The full-scan cost is small (the dictionary is mostly redundant bytes), but the selective-read cost is what page skipping and row-group pruning exist to avoid. Results are correct either way. This is a file layout and read-performance divergence. It showed up when #5677 turned the native writer on by default and the test above started failing its byte bound. The test now writes its fixture on the JVM, so nothing in CI pins this today. ### Describe the potential solution Match parquet-mr's decision closely enough that high-cardinality columns written natively carry no dictionary page. Options, roughly in order of preference: 1. Upstream a compression-ratio fallback to parquet-rs (an opt-in `WriterProperties` setting that abandons the dictionary, and discards it, when the first page shows no saving), then enable it from `build_writer_properties` in `native/core/src/execution/operators/iceberg_write.rs`. 2. Decide per column in Comet, for example by disabling the dictionary for a column once a sample of the first batch shows near-unique values. Cheaper to land, but it's a heuristic that won't match parquet-mr's cut-off exactly. 3. Leave the dictionary on and document the divergence under the accepted differences in `iceberg-writes.md`. Whichever is chosen, add a test that writes a high-cardinality column natively and asserts there is no dictionary page (or a bounded one), so the layout is pinned. ### Additional context Part of #5649 (Phase 5, performance). It should be settled or explicitly accepted before the native writer is on by default (#5644), since it affects every native-written table with unique-ish columns (ids, UUIDs, timestamps). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
