MaxGekk commented on code in PR #56158:
URL: https://github.com/apache/spark/pull/56158#discussion_r3330632359
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala:
##########
@@ -97,6 +99,14 @@ object RowEncoder extends DataTypeErrorsBase {
case TimestampType if SqlApiConf.get.datetimeJava8ApiEnabled =>
InstantEncoder(lenient)
case TimestampType => TimestampEncoder(lenient)
case TimestampNTZType => LocalDateTimeEncoder
+ // Nano timestamp types intentionally do not honor `lenient`: legacy
`java.sql.Timestamp` /
+ // `java.sql.Date` external types are out of scope for nanosecond
precision (SPARK-57033).
+ case t: TimestampNTZNanosType =>
Review Comment:
Yes - this is intentional. The scope of SPARK-57033 is the classic-mode
java.time <-> internal-row conversion (encoders / CatalystTypeConverters /
Dataset roundtrip), so the Connect + Arrow path is deliberately left out
here and
is tracked by follow-up sub-tasks under the umbrella SPARK-56822:
- Arrow type mapping (`ArrowUtils.toArrowType` / `fromArrowType` - the
`unsupportedDataTypeError` you hit): SPARK-57159 "Add Arrow type mapping
for
nanosecond-capable timestamp types".
- The encoder <-> vector dispatch in `ArrowSerializer` / `ArrowDeserializer`
(the `Unsupported Encoder/Vector` RuntimeException) plus the end-to-end
Connect
flow: SPARK-57160 "Add Spark Connect protocol support for
nanosecond-capable
timestamp types and literals" and SPARK-57161 "Convert nanosecond-capable
timestamp types and literals between proto and Catalyst in Spark Connect".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
