andygrove opened a new issue, #7051: URL: https://github.com/apache/datafusion-comet/issues/7051
### What is the problem the feature request solves? Nested types (structs, arrays and maps) in the native Parquet writer and the native Iceberg writer were audited on `main` at c230bf4e98, as #7040 did for the scans. The audit looked at NULL structs, arrays and maps, NULLs inside them, field ids at every level, Parquet statistics and Iceberg manifest metrics. Each write was made twice, once natively and once by Spark with Comet disabled, and the two were compared on the values read back, the Parquet schema and footer, the column statistics, the manifest metrics and the files pruned for nested predicates. Parquet writes ran on Spark 3.4.3, 3.5.9 and 4.1.3, and Iceberg writes on Iceberg 1.5.2, 1.8.1 and 1.11.0. A code review of both writers, arrow-rs 59.3 and iceberg-rust at the current pin ran alongside, and each finding below was reproduced before it was listed. Both writers write nested data correctly. What remains are Spark settings the native Parquet writer ignores, a fallback that keeps writes into `NOT NULL` Iceberg columns off the native path, and one wrong statement in the docs. This epic collects the new issues and the related ones that were already open. Items are listed by priority within each section. ### Correctness: native Parquet writer - [ ] #7047 **The datetime rebase modes are ignored.** Under the Spark 3.4 and 3.5 defaults, the native writer writes pre-1582 dates and timestamps that Spark refuses to write, and Spark then can't read them back. `LEGACY` reads shift them, on every version. - [ ] #7048: `outputTimestampType` and `writeLegacyFormat` are ignored, and Spark's schema footer key is missing. With `TIMESTAMP_MILLIS`, timestamps read back with the microseconds Spark drops. - [ ] #3427: write `org.apache.spark.version` into the footer. #7047 needs it. - [ ] #3425: `INT96` timestamps, the Spark default. ### Performance and coverage: native Iceberg writer - [ ] #7049: writes into `NOT NULL` columns, top-level or nested, fall back on Spark 3.5+ when no exchange sits in front of the writer, because Comet has no native `AssertNotNull`. Native Iceberg writes are on by default (#6664). - [ ] #6959: `uuid` and `unknown` columns fall back, nested ones included. ### Performance: both writers - [ ] #5304: Spark's Parquet writer properties are ignored. For nested rows, the row-count cap gives one row group per file, 239 MB for 100,000 rows of 100-element arrays where Spark writes row groups of about 128 MB ([measured there](https://github.com/apache/datafusion-comet/issues/5304#issuecomment-6217866177)). - [ ] #6918, #6919 and #6920: per-value costs in the parquet-rs writer for nested columns (level encoding, string statistics, dictionary interning and level computation), from #6911. - [ ] #6914: a cheaper `DictionaryChooser` pre-pass for the native Iceberg writer. ### Documentation - [ ] #7050: `iceberg-writes.md` says the float/double null-count difference can't change `IS NULL` pruning. It can, and the native counts are the correct ones (apache/iceberg#17561). ### Related epics - #7040: nested types in the native Parquet and Iceberg scans. - #1625: native Parquet writes. - #5649: production-quality native Iceberg writes. - #6911: performance follow-ups from profiling the end-to-end Iceberg benchmark. - #4764: invalid UTF-8 in native `StringType`, including the writers' input from the JVM. ### Describe the potential solution Each item is a linked issue. Check it here when it closes. ### Additional context **Differences that are not Comet bugs**, listed so they are not refiled: - On Iceberg 1.9 and later, iceberg-java's `value_counts` and `null_value_counts` for a float or double field under a NULL struct leave out the NULL-struct rows. The native writer takes its counts from the Parquet footer, which includes them (#6659, closed). iceberg-java's counts are the wrong ones, because they let `IS NULL` prune files that hold matching rows (apache/iceberg#17561). - parquet-rs normalizes a float or double zero bound to `-0.0` or `+0.0`. That is documented and never prunes more than iceberg-java's bounds. - In a copy-on-write `UPDATE` and in a partitioned write, Spark's `AssertNotNull` projection runs in the JVM in front of a columnar shuffle. The `NOT NULL` check still applies there, and the write stays native. **What matched**: values, Parquet repetition, field ids and column statistics for NULL structs, arrays and maps, empty arrays and maps, and NULLs inside them at every level, in deep and wide schemas. That includes input sliced by `LIMIT` and `OFFSET`, filtered input, and nested values computed by expressions before the write. Parquet field ids at every level, with `spark.sql.parquet.fieldId.write.enabled` on and off. Nested timestamp, decimal, boolean and binary leaves. Iceberg manifest metrics (value, null and NaN counts, lower and upper bounds) under the default and `full` modes on all three Iceberg versions, and on 1.11.0 also under `counts`, `truncate(4)`, per-column overrides on nested fields, `max-inferred-column-defaults` and tiny pages and row groups. Nested partition sources (identity on `s.a`, `bucket`, `truncate` and `days(ts.t)`), a sort order on a nested field, and copy-on-write `DELETE`, `UPDATE` and `MERGE`. Required nested fields under NULL parents written straight from a native scan, and `fixed` fields under NULL structs. Natively written files with pre-1582 values read back correctly by Comet's own scan. **Not covered**: Variant (#5433), partitioned and bucketed native Parquet writes (#3193, #3194), object-store and HDFS outputs, Iceberg format v3 types, encryption, and write throughput. The probes ran against a debug build, and #6911 profiles a release build. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
