alamb commented on issue #26916: URL: https://github.com/apache/arrow/issues/26916#issuecomment-2085672258
This came up on discord recently so moving some info DataFusion supports reading the same parquet file using multiple threads Here is the low level arrow API to do it from @tustvold > You can read row groups in parallel by creating separate readers with different row group selections https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/type.ParquetRecordBatchReaderBuilder.html#method.new_with_metadata This is used by by datafusion automatically. The function that does the parallelization is 1. https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#method.repartitioned 2. implemented for https://docs.rs/datafusion/latest/datafusion/datasource/physical_plan/parquet/struct.ParquetExec.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
