adamreeve commented on code in PR #16351:
URL: https://github.com/apache/datafusion/pull/16351#discussion_r2136730391
##########
datafusion/datasource-parquet/src/file_format.rs:
##########
@@ -1259,9 +1302,14 @@ impl FileSink for ParquetSink {
object_store: Arc<dyn ObjectStore>,
) -> Result<u64> {
let parquet_opts = &self.parquet_options;
- let allow_single_file_parallelism =
+ let mut allow_single_file_parallelism =
parquet_opts.global.allow_single_file_parallelism;
+ if parquet_opts.global.file_encryption_properties.is_some() {
+ // For now, arrow-rs does not support parallel writes with
encryption
+ allow_single_file_parallelism = false;
+ }
Review Comment:
There is an arrow-rs issue for this
(https://github.com/apache/arrow-rs/issues/7359). Maybe we should reference
that in a comment and add a todo note to remove this code once that's fixed and
included in version of the parquet crate being used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
