Jonathancui123 commented on PR #36871: URL: https://github.com/apache/spark/pull/36871#issuecomment-1175592703
> would appreciate it if you summarize any unaddressed comments or concerns. I am fine given that we disable this here by default. @HyukjinKwon I don't see any remaining unaddressed comments - I believe we are ready to merge this PR. Here are the resolutions to potential concerns: - No backwards compatibility concern: `Legacy` parser policy cannot be used with `inferDate`. We can open another ticket to add a `LegacyStrictSimpleDateFormatter` [as Bruce suggested](https://github.com/apache/spark/pull/36871#discussion_r905348742) if necessary. - Performance degradation: Benchmark results ([in the bottom of PR description](https://github.com/apache/spark/pull/36871#issue-1271523396)) show that date inference greatly slows down schema inference for CSVs with dates. This feature is guarded by the CSV Option `inferDate` and is off by default. - Logical errors for strict parsing and converting Date to TimestampNTZ have been fixed and have unit tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
