HyukjinKwon commented on a change in pull request #26027: [SPARK-24540][SQL]
Support for multiple character delimiter in Spark CSV read
URL: https://github.com/apache/spark/pull/26027#discussion_r331732358
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVExprUtils.scala
##########
@@ -79,4 +82,48 @@ object CSVExprUtils {
throw new IllegalArgumentException(s"Delimiter cannot be more than one
character: $str")
}
}
+
+ /**
+ * Helper method that converts string representation of a character sequence
to actual
+ * delimiter characters. The input is processed in "chunks", and each chunk
is converted
+ * by calling [[CSVExprUtils.toChar()]]. A chunk is either:
+ * <ul>
+ * <li>a backslash followed by another character</li>
+ * <li>a non-backslash character by itself</li>
+ * </ul>
+ * , in that order of precedence. The result of the converting all chunks is
returned as
+ * a [[String]].
+ *
+ * <br/><br/>Examples:
+ * <ul><li>`\t` will result in a single tab character as the separator (same
as before)
Review comment:
Sorry if I am rushing to read but why do we need this escaping logic? can't
we just set the option as `"\t"` instead of `"""\t"""`?
or .. is something Univocity expects? If this is the case, it would be
easier to review if you link Univocity side's doc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
