stevomitric opened a new pull request, #59575: URL: https://github.com/apache/spark/pull/59575
### What changes were proposed in this pull request? Implicit type coercion no longer narrows a `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` value (p in 7..9) when a function does not support the nanosecond types: - `TypeCoercion.implicitCast` (non-ANSI) no longer casts a nanos timestamp to a lower-precision timestamp (the microsecond `TIMESTAMP` / `TIMESTAMP_NTZ`, `AnyTimestampType`, or a nanos type of lower precision) or to `TIME`. Before, the `(DatetimeType, DatetimeType)` arm accepted any such pair, so a nanos input silently lost its sub-microsecond digits and an NTZ input became LTZ. - `AnsiTypeCoercion.implicitCast` no longer casts a nanos timestamp to `TIME`. `Cast.canANSIStoreAssign` allows it (store assignment keeps it), and the ANSI implicit-cast rule used that predicate directly. Precision narrowing was already blocked there by SPARK-57303. - The `TypeCollection` fallback in both modes skips `DATE` for a nanos input, so it no longer picks the first `DATE` member and drops the time of day. Both helpers live in `TypeCoercionHelper` (`isNanosNarrowing`, `pickTypeCollectionCast`) and are shared by the two modes. `DATE` as a direct expected type is unchanged, so date-valued functions (`last_day`, `next_day`, `datediff`, `add_months`, `trunc`, ...) still accept a nanos input. Widening (micro to nanos, p to a higher p, the NTZ/LTZ family rules), string promotion, explicit `CAST`, store assignment, and every microsecond type are unchanged. ### Why are the changes needed? The SPIP (SPARK-56822) parity criterion says no function may silently downgrade a nanos timestamp; a clean error is acceptable, a less precise or wrong-typed result is not. Today: - Non-ANSI: `timestampdiff(MICROSECOND, TIMESTAMP_NTZ '...00.000000999', TIMESTAMP_NTZ '...00.000001001')` returns `1` instead of `0`; `months_between`, `window`, `session_window` (and `window_time`) compute on microseconds. ANSI rejects the same calls, so the two modes disagree. - Both modes: `percentile_approx` / `approx_percentile` / `histogram_numeric` over a nanos column return `DATE` (`2020-01-01`). - ANSI: `percentile` / `median` / `percentile_cont` / `percentile_disc` return `TIME(6)` (`12:00:00.123456`), as do `time_trunc` and `time_diff` on a nanos input. These now fail with `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` in both modes until the functions get native nanos support. ### Does this PR introduce _any_ user-facing change? Yes, for the preview nanosecond timestamp types only (`spark.sql.timestampNanosTypes.enabled`): the calls above fail at analysis instead of returning a less precise or wrong-typed value. The same applies to every other implicit-cast site, so in non-ANSI mode a Scala UDF or a procedure parameter typed as a microsecond timestamp now rejects a nanos argument, as ANSI already did. Microsecond timestamps are unaffected. ### How was this patch tested? - `TypeCoercionSuiteBase` (runs in `TypeCoercionSuite` and `AnsiTypeCoercionSuite`): the nanos cast matrix (micro / lower-precision / TIME targets rejected, DATE / string / widening kept, `TypeCollection` fallback skips DATE and TIME, micro unchanged) and rule tests for `months_between`, `time_trunc`, `percentile_approx` and `last_day`. - `timestamp-ntz-nanos.sql` / `timestamp-ltz-nanos.sql`: the affected functions plus date-valued controls, under ANSI on and off. - Existing `TimestampNanos*` suites, `CastWithAnsiOn/OffSuite`, `DateExpressionsSuite`, the catalyst analysis suites and `SQLQueryTestSuite` pass. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
