xiaonanyang-db commented on code in PR #37933:
URL: https://github.com/apache/spark/pull/37933#discussion_r976108061
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala:
##########
@@ -59,6 +59,11 @@ class CSVInferSchema(val options: CSVOptions) extends
Serializable {
ExprUtils.getDecimalParser(options.locale)
}
+ // Date formats that could be parsed in DefaultTimestampFormatter
+ // Reference: DateTimeUtils.parseTimestampString
+ private val LENIENT_TS_FORMATTER_SUPPORTED_DATE_FORMATS = Set(
Review Comment:
When timestamp format is not specified, the desired behavior is that a
column with mix of dates and timestamps will be inferred as timestamp type if
the lenient timestamp formatter could parse the date strings under the column
as well. To achieve that without bringing other performance concern, we want to
simply check if the dateFormat could be supported by the lenient timestamp
formatter. Does that make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
