stevomitric opened a new pull request, #57993: URL: https://github.com/apache/spark/pull/57993
### What changes were proposed in this pull request? Add end-to-end regression-lock coverage for `DISTINCT` over the nanosecond-precision timestamp types `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p` in `[7, 9]`). New suite `TimestampNanosDistinctSuiteBase` (with `TimestampNanosDistinctAnsiOnSuite` / `TimestampNanosDistinctAnsiOffSuite` subclasses), mirroring `TimestampNanosJoinSuiteBase`. For NTZ and LTZ, across `p` in `[7, 9]` and both whole-stage-codegen modes, it asserts: - DISTINCT removes exact duplicates but keeps two values that share `epochMicros` and differ only within the microsecond, and keeps exactly one NULL (unlike an equi-join, where NULL never matches NULL); - a `UNION` of two different nanos precisions widens the column to the higher precision (`findWiderDateTimeType`) and preserves the distinction. Also extends the golden files `timestamp-ntz-nanos.sql` / `timestamp-ltz-nanos.sql` with a DISTINCT case. ### Why are the changes needed? DISTINCT over a nanosecond-timestamp column already works today -- it rides on the generic nanos hashing/equality implemented in SPARK-57103 (`Murmur3Hash` / `XxHash64` / `HiveHash` over the carrier's `epochMicros: Long` and `nanosWithinMicro: Short in [0, 999]`) -- but lacked dedicated coverage. These tests lock the regression: if the nanos hash or equality path is later broken, sub-microsecond-distinct values collapse and the tests fail loudly. No production change. ### Does this PR introduce _any_ user-facing change? No. Test-only. ### How was this patch tested? `TimestampNanosDistinctAnsiOnSuite` / `TimestampNanosDistinctAnsiOffSuite` (16 tests, all pass) and the regenerated golden SQL files. ### Was this patch authored or co-authored using generative AI tooling? Co-Authored-By: Claude Opus 4.8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
