Re: [PR] fix: pass `quote` parameter to CSV writer [datafusion]

2024-05-26 Thread via GitHub


alamb commented on code in PR #10671:
URL: https://github.com/apache/datafusion/pull/10671#discussion_r1615150442


##
datafusion/sqllogictest/test_files/csv_files.slt:
##
@@ -161,3 +161,44 @@ physical_plan
 01)SortPreservingMergeExec: [int_col@0 ASC NULLS LAST]
 02)--SortExec: expr=[int_col@0 ASC NULLS LAST], preserve_partitioning=[true]
 03)CsvExec: file_groups={2 groups: 
[[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/csv_files/csv_partitions/1.csv],
 
[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/csv_files/csv_partitions/2.csv]]},
 projection=[int_col, string_col, bigint_col, partition_col], has_header=false
+
+
+# ensure that correct quote character is used when writing to csv
+statement ok
+CREATE TABLE table_with_necessary_quoting (
+  int_col INT,
+  string_col TEXT,
+) AS VALUES
+(1, 'e|e|e'),
+(2, 'f|f|f'),
+(3, 'g|g|g'),
+(4, 'h|h|h');
+
+# quote is required because `|` is delimiter and part of the data

Review Comment:
   I double checked the contents of the file look good
   ```shell
   andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion$ cat 
./datafusion/sqllogictest/test_files/scratch/csv_files/table_with_necessary_quoting.csv
   int_col|string_col
   1|~e|e|e~
   2|~f|f|f~
   3|~g|g|g~
   4|~h|h|h~
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org



Re: [PR] fix: pass `quote` parameter to CSV writer [datafusion]

2024-05-26 Thread via GitHub


alamb merged PR #10671:
URL: https://github.com/apache/datafusion/pull/10671


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org



[PR] fix: pass `quote` parameter to CSV writer [datafusion]

2024-05-25 Thread via GitHub


DDtKey opened a new pull request, #10671:
URL: https://github.com/apache/datafusion/pull/10671

   
   
   ## Which issue does this PR close?
   
   Closes #10670
   
   ## Rationale for this change
   
   
   
   ## What changes are included in this PR?
   
   
   
   ## Are these changes tested?
   
   `sqllogictest` has been added which fails without these changes
   
   ## Are there any user-facing changes?
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org