joosthooz commented on PR #13709: URL: https://github.com/apache/arrow/pull/13709#issuecomment-1199314506
Thank you for the comments and suggestions. I implemented Weston's suggestion and added an `encoding` field to `CsvFileFormat` instead of the added parameter to `dataset()`. This works, but I dislike it a lot. The field is a duplicate to the same one in `ReadOptions`. So when using `read_csv`, users need to use the field in `read_options`, but when using `dataset`, they need to use the field in `format`. They can also use both of them, 1 is silently discarded. Here's what this looks like: ``` >>> fo = ds.CsvFileFormat(default_fragment_scan_options=ds.CsvFragmentScanOptions(read_options=csv.ReadOptions(encoding='iso-8259')), encoding='cp1252') >>> fo.default_fragment_scan_options.read_options.encoding 'utf8' >>> fo.encoding 'cp1252' ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
