stevomitric opened a new pull request, #57699: URL: https://github.com/apache/spark/pull/57699
### What changes were proposed in this pull request? This PR adds bidirectional proto <-> Catalyst conversion for the nanosecond-capable timestamp types `TimestampNTZNanosType(p)` and `TimestampLTZNanosType(p)` (precision `p` in [7, 9]) in Spark Connect's shared connect-common converters, so the protocol messages added by SPARK-57160 become usable by both the JVM Connect client and server. The Connect Types Framework delegates all proto DataType/Literal conversion through `ConnectTypeOps`, so the change is small and mirrors the existing `TimeType` integration: - New `TimestampNanosTypeConnectOps` (a shared base plus `TimestampNTZNanosTypeConnectOps` / `TimestampLTZNanosTypeConnectOps`) implementing the proto DataType and Literal side of `ConnectTypeOps`. The external java.time value is `LocalDateTime` (NTZ) / `Instant` (LTZ), matching the server-side `TypeOps` and `RowEncoder`. The literal carries the physical value as `epoch_micros` + `nanos_within_micro` (mirroring the Catalyst `TimestampNanosVal`), because a single int64 of nanoseconds cannot span the supported 0001..9999 year range. - `ConnectTypeOps`: register both types at the three proto-dispatch points (`apply`, `opsForKindCase`, `literalCaseToKindCase`). - `LiteralValueProtoConverter`: add the two `isCompatible` literal/type cases. Arrow IPC serialization is out of scope for this sub-task, so the ops is not registered in the Arrow dispatch and its Arrow methods throw. The `spark.sql.timestampNanosTypes.enabled` feature flag continues to be enforced on the server path via `TypeUtils.failUnsupportedDataType` (the same way `TimeType` is gated), so no extra guard is added in connect-common. ### Why are the changes needed? The proto sub-task (SPARK-57160) added the wire surface only. Without these converters, schema responses, casts, UDF input/output types, and literal expressions that contain nanosecond timestamps fail in Connect with `CONNECT_INVALID_PLAN.DATA_TYPE_UNSUPPORTED_*`. These converters unblock both the client and the server. Sub-task of SPARK-56822. ### Does this PR introduce _any_ user-facing change? No. The types remain gated behind `spark.sql.timestampNanosTypes.enabled` (default off in production). ### How was this patch tested? New tests in `LiteralExpressionProtoConverterSuite` cover DataType round-trips across precisions 7-9, NTZ and LTZ literal round-trips (boundary, pre-epoch, and sub-microsecond values), the two-component literal encoding, and rejection when the feature flag is disabled. - `connect/testOnly *LiteralExpressionProtoConverterSuite` -- 49 tests passed (5 new) - `connect-client-jvm/testOnly *ColumnNodeToProtoConverterSuite` -- 18 tests passed - scalastyle (main + test) -- 0 errors ### Was this patch authored or co-authored using generative AI tooling? Co-authored-by: Claude Code (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
