Thanks Antione for the pointers. I switched to using io::BufferReader and that fixed the issue. Newline options also work fine. It is an extra copy for me, but for now I am good.
On Wed, Mar 9, 2022 at 7:46 AM Antoine Pitrou <anto...@python.org> wrote: > On Mon, 7 Mar 2022 11:52:02 -0800 > HK Verma <hkve...@gmail.com> wrote: > > > Thanks Antoine. Yes I have newlines_in_values set to false. Other configs > > also look ok. > > However I do have rows with less number of columns than the specified > > numbers in convert options in column types. I have my own > > invalid_row_handler where I currently skip these rows. > > It looks like the parser is doing a quick pass and splitting the blocks > > using the new line separator and parsing them in parallel. This seems > > similar to spark csv multiline option. To avoid this case, I did set > > newlines_in_values to false and also checked if newlines are not there in > > any of the fields. > > No, you should check `newlines_in_values` to true and see if the error > goes away. > > Regards > > Antoine. > > >
