andygrove opened a new issue, #5304:
URL: https://github.com/apache/datafusion-comet/issues/5304
### What is the problem the feature request solves?
`ParquetWriterExec` builds its `WriterProperties` with compression and
nothing else (`native/core/src/execution/operators/parquet_writer.rs`):
```rust
let props = WriterProperties::builder()
.set_compression(compression)
.build();
```
Every other Parquet write knob Spark exposes is therefore silently ignored,
and files come out with arrow-rs defaults rather than the ones the user
configured:
- `parquet.block.size` (row group size) — tracked as an unchecked item on
#2967
- `parquet.page.size` / `parquet.page.row.count.limit`
- `parquet.enable.dictionary` and dictionary page size
- `spark.sql.parquet.writer.version` (`PARQUET_1_0` / `PARQUET_2_0`)
- statistics level, and bloom filter settings
- `spark.sql.files.maxRecordsPerFile` (needs file rolling, so partly
separate)
Row group size is the one with real consequences: the default row group size
decides read parallelism and memory for every downstream consumer, and a user
who set `parquet.block.size` gets no error, just a differently shaped file.
This also matters beyond the V1 write path — #4658 splits Iceberg writes
into writer + committer operators with the stated goal of writing data files
natively next, and Iceberg data files are read by engines other than Spark.
### Describe the potential solution
Thread the resolved options through the `ParquetWriter` protobuf message and
set them on `WriterProperties`. Spark's `ParquetOptions` /
`ParquetWriteSupport` are the reference for precedence and defaults.
Related: #2814 (compression settings), #3425 (INT96), #3427 (Spark version
in footer metadata).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
