stevomitric opened a new pull request, #59785: URL: https://github.com/apache/spark/pull/59785
### What changes were proposed in this pull request? Subtraction with a nanosecond-precision timestamp operand (`TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)`, p in [7, 9]) now fails analysis with the new `UNSUPPORTED_FEATURE.TIMESTAMP_NANOS_SUBTRACTION` error, instead of returning a microsecond-grid day-time interval (SPARK-57832). This covers nanos - nanos, mixed micro/nanos, nanos/DATE in either order, and legacy interval mode. The error points to `unix_nanos(end) - unix_nanos(start)` for the exact difference, or to casting both operands to the micro types. The check is a new `NanosTimestampSubtractionCheck`, run from `CheckAnalysis` and from the single-pass resolver's resolution checks. The as-of join's internal ordering distance (`AsOfJoin.orderExpression`) is exempt, so DataFrame `joinAsOf` over nanosecond keys keeps working. ### Why are the changes needed? The result silently drops sub-microsecond digits (`'…05.123456789' - '…05.000000111'` gives `0.123456`). Day-time intervals may later gain fractional-seconds precision (`INTERVAL DAY TO SECOND(p)`, as in Oracle, Snowflake and Trino), and then `ts(p1) - ts(p2)` should return `INTERVAL DAY TO SECOND(max(p1, p2))`. Changing a result type after release is hard, so it's better to fail now (agreed with @cloud-fan). ### Does this PR introduce _any_ user-facing change? Yes, for nanosecond timestamps, which are behind `spark.sql.timestampNanosTypes.enabled`. Subtraction with a nanosecond operand now fails analysis instead of returning a microsecond-grid interval. Microsecond timestamp subtraction is unchanged. ### How was this patch tested? - Golden cases in `timestamp-ntz-nanos.sql` / `timestamp-ltz-nanos.sql` for every operand mix, plus the `unix_nanos` and cast workarounds (both analyzers agree). - New tests in `TimestampNanosFunctionsSuiteBase` (SQL and Column API, legacy interval mode) and `TimestampNanosJoinSuiteBase` (DataFrame `joinAsOf` over nanos keys, with and without sort-merge), ANSI on and off. - `SortMergeAsOfJoinSuite`, `DataFrameAsOfJoinSuite`, `AsOfJoinSortMergeSQLSuite`, `TypeCoercionSuite`, `AnsiTypeCoercionSuite`, `DateExpressionsSuite`, `SparkThrowableSuite` and the timestamp/time/asof golden files pass. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
