MassivePizza opened a new issue, #23949: URL: https://github.com/apache/datafusion/issues/23949
### Is your feature request related to a problem or challenge? Writing Parquet V2 can benefit from [set_data_page_v2_compression_ratio_threshold](https://docs.rs/parquet/58.3.0/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_data_page_v2_compression_ratio_threshold), which is not exposed in Datafusion. ### Describe the solution you'd like Expose `data_page_compression_ratio_threshold` on `datafusion_common::config::ParquetOptions` and `ParquetColumnOptions` so it can be passed along to `WriterPropertiesBuilder`. This primarily enables `datafusion_datasource_parquet::file_format::ParquetFormat` to use this setting. ### Describe alternatives you've considered We are evaluating if we should always disable compression for sorted delta-encoded columns. This can easily be done with `ParquetColumnOptions.compression` today. We can also create a wrapper `ParquetSink` to set more options on the `WriterPropertiesBuilder` but that's quite a painful workaround for little gain. ### Additional context We experience poor compression ratios (<5%) with Zstd for delta-encoded columns of timestamps and other types. Disabling compression with a threshold would be ideal to avoid the decompression overhead at query time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
