wyatt-herkamp commented on PR #27021: URL: https://github.com/apache/datafusion/pull/27021#issuecomment-6659946398
Thanks for the review @xudong963. Addressed in 70ec58c and rebased on main: - Header discovery now follows csv-core: a leading UTF-8 BOM is stripped, empty records are skipped (both `\r` and `\n` when no terminator is set), and comment lines end at `\n`. Whitespace-only lines are still records, as in the parser. - Regression tests: unit tests on `find_header_line` for BOM, blank lines and comments, plus an integration test with a BOM + comment + blank lines before the header across local, chunked and repartitioned reads. - `CsvReadOptions::match_columns_by_header` exposes the option on `read_csv`, with a test. - The new public fields are documented in the 56.0.0 upgrade guide. I don't have permission to add the `api change` label, so could you add it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
