Andy Grove created ARROW-11012: ---------------------------------- Summary: [Rust] [DataFusion] Make write_csv and write_parquet concurrent Key: ARROW-11012 URL: https://issues.apache.org/jira/browse/ARROW-11012 Project: Apache Arrow Issue Type: Improvement Components: Rust - DataFusion Reporter: Andy Grove
ExecutionContext.write_csv and write_parquet currently iterate over the output partitions and execute one at a time and write the results out. We should run these as tokio tasks so they can run concurrently. This should, in theory, help with memory usage when the plan contains repartition operators. We may want to add a configuration option so we can choose between serial and parallel writes? -- This message was sent by Atlassian Jira (v8.3.4#803005)