stevomitric opened a new pull request, #58418: URL: https://github.com/apache/spark/pull/58418
### What changes were proposed in this pull request? Exposes the nanosecond-capable timestamp types `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p` in [7, 9]) in PySpark. Before this change `python/pyspark/sql/types.py` defined only the microsecond singletons, so any DataFrame whose schema contained one of these types failed on the Python side even though the JVM and the Spark Connect protocol already supported them. - `python/pyspark/sql/types.py`: adds `TimestampNTZNanosType(precision)` and `TimestampLTZNanosType(precision)` (plus the shared `AnyTimestampNanoType` base, which is not exported), with precision validation, `simpleString` / `jsonValue` / `__repr__`, and `toInternal` / `fromInternal`. Registers the parameterized JSON type names, mirroring `DataType.parseDataType` in `sql/api`: precision 6 maps to the standard microsecond type, 7-9 to the nanosecond types, and any other precision is rejected. Also registers the two types in `_acceptable_types` (used by `_make_type_verifier`, so `createDataFrame` accepts `datetime.datetime` values) and in `_get_jvm_type_name` (so `printSchema()` renders `timestamp_ntz(9)` rather than a name derived from the class). - `python/pyspark/sql/connect/types.py`: converts the two types to and from the Connect `DataType` proto in both directions, treating an omitted `precision` as 9 per types.proto. - `sql/api/.../types/ops/TimestampNanosTypeApiOps.scala`: implements the Types Framework Python-interop hooks (`needConversionInPython`, `makeFromJava`) so values round-trip over Py4J. Without these, `EvaluatePython.makeFromJava` fell through to its catch-all and silently produced NULL for every nanosecond column. - `sql/core/.../EvaluatePython.scala`: adds the reverse direction, converting the internal `TimestampNanosVal` to epoch microseconds. Without it the value reached the pickler as a raw `TimestampNanosVal`, which has no registered pickler. - `python/pyspark/errors/error-conditions.json`: adds `INVALID_TIMESTAMP_PRECISION`, worded to match the JVM error condition of the same name. The external Python value is `datetime.datetime`, which is microsecond-resolution, so the Py4J protocol carries epoch microseconds and sub-microsecond digits are truncated at the Python boundary in both directions. This mirrors the shipped `TimeType` behaviour and is the microsecond-only Python/UDF limitation already documented by SPARK-57808; the stored value keeps full precision. Type inference is unchanged: a bare `datetime.datetime` still infers microsecond `TimestampType`, and the nanosecond types are reachable only through an explicit schema. Arrow and pandas value conversion (`toPandas`, `createDataFrame` from pandas, and therefore the Spark Connect data path) is deliberately not included here and remains follow-up work; `to_arrow_type` continues to reject these types. ### Why are the changes needed? Without Python type classes, a `TIMESTAMP(9)` column cannot be read or written from PySpark at all: `proto_schema_to_pyspark_data_type` raised `UNSUPPORTED_OPERATION` for the Connect schema, and reading `df.schema` failed because the parameterized JSON type name had no Python parser. This is the last missing client for the umbrella SPARK-56822. ### Does this PR introduce _any_ user-facing change? Yes. `TimestampNTZNanosType` and `TimestampLTZNanosType` are new public types in `pyspark.sql.types`. They are only reachable via an explicit schema or a nanosecond-typed query result, and the server keeps them behind the `spark.sql.timestampNanosTypes.enabled` preview flag, so no existing behaviour changes. ### How was this patch tested? New tests in `python/pyspark/sql/tests/test_types.py`: - `DataTypeTests`: precision validation (7-9 accepted; -1/0/5/6/10 rejected with `INVALID_TIMESTAMP_PRECISION`), string representations including `printSchema()` rendering, equality / hashing / pickling, JSON parsing across precisions including the 6 -> microsecond mapping and the rejected precisions, nested array/map/struct JSON round-trip, and `toInternal` / `fromInternal` agreement with the microsecond types. - `DataTypeVerificationTests`: accepted and rejected values for both types. - `TypesTestsMixin.test_timestamp_nanos_type`: DDL parse agreement with the JVM, plus a `createDataFrame` / `collect` round-trip with nulls, and a check that a value stored at nanosecond precision still renders 9 fractional digits server-side while truncating to microseconds when collected as a `datetime`. New test in `python/pyspark/sql/tests/connect/test_connect_plan.py`: DataType proto round-trip for both types across precisions, nested, and with `precision` omitted. ### Was this patch authored or co-authored using generative AI tooling? Co-authored: Claude Opus 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
