navaneethsujith09 commented on PR #12410:
URL: https://github.com/apache/gluten/pull/12410#issuecomment-4917855909
@taiyang-li To answer this question, there were two main reasons for me to
create a new function. While both toDate(1) and toDate32(1) return "1970-01-02"
so that they both look equivalent for small values. ToDate and ToDate32 has
different semantics at the edges like:
- toDate is a 16-bit type and can’t represent dates before 1970
- toDate reinterprets large numbers as unix timestamps in
seconds, not days
- toDate32 clamps out-of-range values to its supported range
[1900-01-01, 2299-12-31]
So for out-of-range inputs, ClickHouse builtins can silently return the
wrong date instead of Spark’s result. Our sparkDateFromUnixDate just passes the
day count through unchanged, which preserves Spark parity. Also, I checked
soupam05's PR on this issue and he also did it in a similar way. Hope that
helps!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
