[
https://issues.apache.org/jira/browse/SPARK-57161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uroš Bojanić resolved SPARK-57161.
----------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 57699
[https://github.com/apache/spark/pull/57699]
> Convert nanosecond-capable timestamp types and literals between proto and
> Catalyst in Spark Connect
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-57161
> URL: https://issues.apache.org/jira/browse/SPARK-57161
> Project: Spark
> Issue Type: Sub-task
> Components: Connect, SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Assignee: Stevo Mitric
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> h2. What
> Implement the proto <-> Catalyst conversion for {{TimestampNTZNanosType(p)}}
> and
> {{TimestampLTZNanosType(p)}} (p in [7, 9]) in Spark Connect's shared
> {{connect-common}}
> converters, so the protocol messages added in the proto sub-task become
> usable by both the
> JVM Connect client and the Connect server.
> Parent: SPARK-56822. Depends on the Spark Connect protocol sub-task (proto
> definitions for
> the nanos timestamp data types and literals).
> h2. Why
> The proto sub-task only adds wire surface; nothing yet maps those messages
> to/from Catalyst
> {{DataType}} and {{Literal}}. Until these converters exist, schema responses,
> casts, UDF
> input/output types, and literal expressions containing nanosecond timestamps
> fail in
> Connect with {{CONNECT_INVALID_PLAN.DATA_TYPE_UNSUPPORTED_*}}. These
> converters are shared
> by the JVM client and the server, so this unblocks both at once.
> The Catalyst physical value is
> {{org.apache.spark.unsafe.types.TimestampNanosVal}}
> ({{epochMicros: Long}} + {{nanosWithinMicro: Short}} in [0, 999]).
> h2. Scope
> Data type conversion
> * {{sql/connect/common/.../DataTypeProtoConverter.scala}}: both directions
> ({{toCatalystType}} and {{toConnectProtoType}}) for the new
> {{TIMESTAMP_NTZ_NANOS}} /
> {{TIMESTAMP_LTZ_NANOS}} kinds, reading/writing {{precision}}.
> * Prefer the Types Framework path: add {{TimestampNTZNanosTypeConnectOps}} /
> {{TimestampLTZNanosTypeConnectOps}} (mirroring {{TimeTypeConnectOps}}) and
> register them in
> {{sql/connect/common/.../types/ops/ConnectTypeOps.scala}} ({{apply}},
> {{opsForKindCase}}).
> * {{sql/connect/common/.../ProtoDataTypes.scala}}: add builders only if
> needed (parameterized
> types typically build inline, like {{TimeType}}).
> Literal conversion
> * {{sql/connect/common/.../LiteralValueProtoConverter.scala}}: outbound
> ({{toLiteralProtoBuilder}} / {{toLiteralProtoWithType}}) and inbound
> ({{getScalaConverter}}, {{isCompatible}}, {{getProtoDataType}},
> {{toDataType}}) handling for
> the new literal kinds, encoding/decoding {{epochMicros}} +
> {{nanosWithinMicro}} + precision.
> * Register literal hooks in {{ConnectTypeOps}} ({{toLiteralProtoForValue}},
> {{literalCaseToKindCase}}).
> Server-side literal -> Catalyst prerequisite
> * {{sql/catalyst/.../CatalystTypeConverters.scala}}: add cases for the two
> types (via the
> {{TypeOps}} registration) so {{LiteralExpressionProtoConverter}} can build
> Catalyst literals
> from the decoded values. ({{LiteralExpressionProtoConverter}} itself needs no
> type-specific
> change.)
> Guard rails
> * Honor {{spark.sql.timestampNanosTypes.enabled}} on the conversion path,
> consistent with the
> Catalyst DDL parser ({{DataTypeErrors.checkTimestampNanosTypesEnabled()}}).
> h2. Out of scope
> * Proto message definitions and stub regeneration (separate proto sub-task;
> this depends on
> it).
> * Arrow IPC type mapping and Arrow encoders (separate sub-tasks).
> * PySpark client conversion (separate sub-task).
> h2. How this changes tested
> * New round-trip tests: Catalyst {{DataType}} -> proto -> Catalyst for NTZ
> and LTZ nanos
> types across precisions 7-9.
> * Literal round-trip: Catalyst {{Literal}} -> proto {{Expression.Literal}} ->
> Catalyst,
> asserting {{epochMicros}} / {{nanosWithinMicro}} / precision are preserved,
> including
> boundary values and pre-epoch instants; verify NTZ vs LTZ stay distinct via
> {{isCompatible}}.
> * Negative test: conversion is rejected when
> {{spark.sql.timestampNanosTypes.enabled=false}}.
> h2. Does this introduce any user-facing change
> No. The types remain gated behind {{spark.sql.timestampNanosTypes.enabled}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
