corwinjoy commented on code in PR #16351:
URL: https://github.com/apache/datafusion/pull/16351#discussion_r2136721651
##########
datafusion/datasource-parquet/src/file_format.rs:
##########
@@ -1259,9 +1302,14 @@ impl FileSink for ParquetSink {
object_store: Arc<dyn ObjectStore>,
) -> Result<u64> {
let parquet_opts = &self.parquet_options;
- let allow_single_file_parallelism =
+ let mut allow_single_file_parallelism =
parquet_opts.global.allow_single_file_parallelism;
+ if parquet_opts.global.file_encryption_properties.is_some() {
+ // For now, arrow-rs does not support parallel writes with
encryption
+ allow_single_file_parallelism = false;
+ }
Review Comment:
We noted this problem in the original PR and @rok is working on it. Would be
nice to have a tracking issue for this @adamreeve. Noted here:
https://github.com/apache/arrow-rs/pull/7111/files#r2015196618
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
