Ziheng Wang created ARROW-17529: ----------------------------------- Summary: Clean up how the CSV reader handles the first buffer Key: ARROW-17529 URL: https://issues.apache.org/jira/browse/ARROW-17529 Project: Apache Arrow Issue Type: Improvement Components: C++, Python Reporter: Ziheng Wang Assignee: Ziheng Wang
Currently how the CSV reader handles the first block in the CSV is not great. In fact I think the first block is read multiple times. First in the Peek in file_csv.cc and then in the InitFromBlock in the OpenReaderAsync in reader.cc This could be problematic if the first block is pretty big, and also delays the synchronous opening of a dataset. Possible solution is to use a smaller block size for the peek in file_csv.cc since you don't need to read the entire block to GetConvertOptions. So we could really just have another option in reader_options that's first_peek_size or something like that. -- This message was sent by Atlassian Jira (v8.20.10#820010)