stevomitric opened a new pull request, #58694: URL: https://github.com/apache/spark/pull/58694
### What changes were proposed in this pull request? Follow-up to SPARK-57462, which added the PySpark `TimestampNTZNanosType` / `TimestampLTZNanosType` classes and the classic (Py4J) value path but deliberately deferred the Arrow / pandas value path: `to_arrow_type` rejected these types, and `DataFrame.toPandas`, `SparkSession.createDataFrame` from a pandas `DataFrame`, and the whole Spark Connect data path raised `UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION`. This change wires the two types through the Python Arrow path. The JVM side already supports them (`ArrowUtils` maps them to an Arrow `Timestamp(NANOSECOND)` field plus a `SPARK::timestampNanos::precision` metadata tag, with native `ArrowWriter` writers and `ArrowColumnVector` accessors). The values are carried as an Arrow `timestamp[ns]` (a 64-bit count of nanoseconds since the epoch), so full nanosecond precision is preserved (pandas `datetime64[ns]`); values outside the `datetime64[ns]` range are out of range for this path, consistent with the JVM `ArrowWriter`, which throws on overflow. ### Why are the changes needed? Without this, a nanosecond-typed column cannot be read into pandas (`toPandas`) or written from a pandas `DataFrame`, and because the Spark Connect data path is entirely Arrow-based, cannot be collected at all over Spark Connect. ### Does this PR introduce _any_ user-facing change? Yes, when the `spark.sql.timestampNanosTypes.enabled` preview flag is on (it is off by default). `DataFrame.toPandas`, `SparkSession.createDataFrame` from a pandas `DataFrame`, and the Spark Connect data path now accept `TimestampNTZNanosType` / `TimestampLTZNanosType` and preserve full nanosecond precision, where they previously raised `UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION`. ### How was this patch tested? - Rewrote the classic `test_timestamp_nanos_type_arrow_conversion_unsupported` test into a positive round-trip `test_timestamp_nanos_type_arrow_conversion` that asserts a value with nine fractional-second digits survives `toPandas` (pandas `Timestamp.nanosecond == 789`) and a pandas -> `createDataFrame` -> Spark -> pandas round-trip, for both the NTZ and LTZ types. - Un-skipped the now-supported nanosecond data-path tests in the Spark Connect parity suite (`connect/test_parity_types.py`) and replaced the "unsupported" Connect test with a positive `test_timestamp_nanos_type_connect_data_path`; the classic-only Py4J-UDF and map-key tests remain skipped there. - Ran the new test against a live `SparkSession` and confirmed it passes, and confirmed the existing `TimestampType` / `TimestampNTZType` / `TimeType` Arrow round-trips are unchanged. ### Was this patch authored or co-authored using generative AI tooling? Co-Authored-By: Claude Opus 4.8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
