d80tb7 commented on pull request #31399: URL: https://github.com/apache/spark/pull/31399#issuecomment-770271817
> I would create a dedicated date/timestamp formatters that don't recognize special values instead of checking of every input string whether it is a special value or not. > Hi- I considered this approach but I was a bit put off as the code actually delegates creating the date/timestamp formatter to a factory which branches off into multiple different classes depending on whether we're in legacy mode etc. We could update the factory such that it knows how to create formatters that don't recognize special values but that would be a more invasive change. Let me know if that's the way you'd like to go and I'll raise a suitable PR. Also I know it's a technicality, but the change here doesn't actually check every input string for special timestamps, just those that successfully parse into a timestamp. > Could you elaborate more in PR description why special date/timestamp values in partition cause problems. Sure ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
