stevomitric opened a new pull request, #59579: URL: https://github.com/apache/spark/pull/59579
### What changes were proposed in this pull request? This PR hardens range and overflow handling for `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (p in [7, 9]). The representable range does not change: it stays the same as for the microsecond types. - `DateTimeUtils.timestampNanosToEpochNanos`, the int64 epoch-nanosecond pack used by the Parquet INT64, Avro and Arrow writers and by Parquet filter pushdown, is now exact down to the int64 minimum. `epochMicros * 1000` overflowed for `epochMicros = -9223372036854776` even when the final value fits, so `1677-09-21 00:12:43.145224192` through `.145224999` UTC were rejected with `DATETIME_OVERFLOW`, although the error message names `1677-09-21T00:12:43.145224192Z` as the supported minimum. - `timestampNanosAddDayTime` / `timestampNanosAddMonths` (`ts + INTERVAL ...` on nanos timestamps) raise `DATETIME_OVERFLOW` when the result leaves the epoch-microsecond range, instead of a raw `java.lang.ArithmeticException: long overflow`. This matches `timestampadd`, which already wraps its overflow the same way. - The `unix_nanos` comment no longer claims `DECIMAL(21, 0)` holds every value: it covers the 0001-9999 calendar range, and values more than ~31,000 years from the epoch raise `NUMERIC_VALUE_OUT_OF_RANGE`. - `docs/sql-ref-datatypes.md` documents the range of the nanos types, the narrower int64 epoch-nanosecond range of Parquet, Avro and Arrow, and the overflow behavior. - Boundary tests at the calendar limits (`0001-01-01 00:00:00`, `9999-12-31 23:59:59.999999999`), the int64 epoch-nanosecond limits (`1677-09-21 00:12:43.145224192`, `2262-04-11 23:47:16.854775807`) and the epoch-microsecond limits, for parse, format, cast, interval and `timestampadd` arithmetic, `unix_nanos`, and the Parquet/Avro/Arrow writes. ### Why are the changes needed? The SPIP (SPARK-56822, Q6) lists range and overflow as a risk, with these mitigations: document the representable range, enforce bounds where values are narrowed, and test edge instants. Before this PR the int64 sinks rejected valid values at the int64 minimum, nanos interval overflow escaped as a raw JVM exception, and the range wasn't documented. Parsing and casting already behave like the microsecond types, because they share the parser: years after 9999 are accepted, and a value outside the epoch-microsecond range gives `NULL`, or `CAST_INVALID_INPUT` under ANSI mode. This PR keeps that parity and does not add a 0001-9999 check, which would make the nanos types stricter than `TIMESTAMP`. ### Does this PR introduce _any_ user-facing change? Yes, for the preview nanosecond timestamp types only, which are not in a released version. The microsecond types are unchanged. - Writing a nanos value from `1677-09-21 00:12:43.145224192` to `1677-09-21 00:12:43.145224999` UTC to Parquet, Avro or Arrow now succeeds. Before, it failed with `DATETIME_OVERFLOW`. - `ts + INTERVAL ...` past the epoch-microsecond range (around year 294247) raises `DATETIME_OVERFLOW` instead of `java.lang.ArithmeticException: long overflow`. ### How was this patch tested? - New catalyst tests in `DateTimeUtilsSuite`, `TimestampNanosParseSuite`, `CastSuiteBase` / `CastWithAnsiOnSuite` / `CastWithAnsiOffSuite`, and `DateExpressionsSuite`: the boundaries above for parse, format, cast, interval add, `timestampadd` and `unix_nanos`, in both ANSI modes, interpreted and codegen. - New sink tests at the exact int64 limits and one nanosecond past them: `ParquetTimestampNanosSuite` (including pushdown), `TimestampNanosParquetOpsSuite`, `AvroSuite` and `ArrowWriterSuite`. - The touched suites, the `TimestampNanos*` suites and the nanos golden files in `SQLQueryTestSuite` pass, and scalastyle is clean. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
