alamb opened a new issue, #6325: URL: https://github.com/apache/arrow-datafusion/issues/6325
### Is your feature request related to a problem or challenge? As part of having a great "out of the box" experience it is important to use as many cores as possible in DataFusion. Given modern consumer laptops have 8-16 cores using multiple cores can literally translate to an order of magnitude faster performance. While DataFusion offers the ability to read partitioned datasets (aka when the input is in multiple files), often, especially for initially testing out the tool, people will simply running queries on their existing CSV or JSON datasets and it will be relatively slow. We already have the great `datafusion.optimizer.repartition_file_scans` option (see [docs](https://arrow.apache.org/datafusion/user-guide/configs.html)) -- added by @korowa in https://github.com/apache/arrow-datafusion/pull/5057 (👋 !) which uses multiple cores to decode the parquet files in parallel. I would like a similar feature for CSV files ### Describe the solution you'd like One basic approach (following what @korowa did for Parquet) would be: 1. If the `datafusion.optimizer.repartition_file_scans` option is set, divide the file into even (byte) sized contiguous blocks, probably with some lower limit (like 1MB) 3. Update [CsvExec](https://github.com/apache/arrow-datafusion/blob/8a25953182b361a58c56384129bf57f32aa2dbb1/datafusion/core/src/physical_plan/file_format/csv.rs#L53) to process partitions using those subsets of the viles Notes: Given the vagaries of CSV (e.g. unescaped quoted newlines) it is likely impossible to parallelize CSV reading for all possible files. I think this is fine, and as long as we can turn off the reading in parallel it is better to have faster out of the box query performance for 99.99% of the queries than handle bizzare CSV files always Care will be required to make sure all records are read exactly once, given the partition splits will likely be in the middle of rows. One idea for parsing a partition (`offset`, `len`): 1. Start CSV parsing the data starting *after* finding the next newline after the `offset` bytes 2. Continue CSV parsing until the newline *after* the `offset + len` byte ``` 0 A,1,2,3,4,5,6,7,8,9\n 20 A,1,2,3,4,5,6,7,8,9\n 40 A,1,2,3,4,5,6,7,8,9\n ◀─ ─ ─ ─ ─ ─ ─ ─ 60 A,1,2,3,4,5,6,7,8,9\n │ 80 A,1,2,3,4,5,6,7,8,9\n 100 A,1,2,3,4,5,6,7,8,9\n │ Byte Offset Lines of CSV Data │ (in this case 20 bytes per line) Split at byte 50 is in the middle of this line ``` ### Describe alternatives you've considered _No response_ ### Additional context @kmitchener noticed the same thing in: https://github.com/apache/arrow-datafusion/issues/5205 The duckdb implementation of a similar feature may offer some inspiration: https://github.com/duckdb/duckdb/pull/5194 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
