arina-ielchiieva commented on issue #1873: DRILL-6096: Provide mechanism to configure text writer configuration URL: https://github.com/apache/drill/pull/1873#issuecomment-544469217 @paul-rogers thanks for the code review, addressed code review comments, force-pushed since there were minor changes in the code. Regarding design, the aim of this Jira was just to fix text writer to write proper text files: before if column contained field separator, field was not enclosed in the quotes, thus we were writing text files which Drill could not read. Now when user indicates write format using session option (this is common approach for all formats), Drill produces text files, it can read back. Basically, if user has configured format plugin: ``` "formats": { "csvh": { "type": "text", "extensions": [ "csvh" ], "lineDelimiter": "\n", "fieldDelimiter": ",", "extractHeader": true } }, ``` Drill will be able to read and write such text files correctly. Same approach is used for `parquet`, `json`. All user needs to do is to indicate write format using session option: `alter session set `store.format` = 'csvh';` (`parquet`, `json`). I am not saying this is ideal and we might need to reconsider such writing approach but I guess not in the scope of Jira since such re-design would touch all file writers.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
