wyatt-herkamp commented on PR #27021: URL: https://github.com/apache/datafusion/pull/27021#issuecomment-6745080989
Thanks @xudong963, fixed in 0f5dd1f. Header discovery now parses the header record with a `csv_core::Reader` configured like arrow's decoder, so quoting, escapes, BOM, blank lines and comments match the parser exactly. Added `csv_match_columns_by_header_with_quoted_terminator` (local + chunked reads) using `"x;y",c,a;` with `;` as the terminator, plus unit tests for quoted terminators, escapes and incomplete input. One correction to my earlier reply: with a non-`\n` terminator, csv-core doesn't end a comment at `\n` (it runs to the end of the input), and header discovery now follows that too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
