MaxGekk opened a new pull request, #56158:
URL: https://github.com/apache/spark/pull/56158
### What changes were proposed in this pull request?
This PR wires `java.time.LocalDateTime` / `java.time.Instant` through the
encoder, converter and Dataset stack for the nanosecond-capable timestamp types
`TimestampNTZNanosType(p)` and `TimestampLTZNanosType(p)` introduced by
SPARK-56981, so that `spark.createDataFrame(rows, schema).collect()` over a
schema declaring a nanos timestamp column preserves full nanosecond precision
end-to-end.
Concretely:
- `SparkDateTimeUtils` (mixed into `DateTimeUtils`): four new helpers
- `localDateTimeToTimestampNanos(LocalDateTime): TimestampNanosVal`
- `timestampNanosToLocalDateTime(TimestampNanosVal): LocalDateTime`
- `instantToTimestampNanos(Instant): TimestampNanosVal`
- `timestampNanosToInstant(TimestampNanosVal): Instant`
Sub-microsecond digits (`getNano % 1000`) are kept as `nanosWithinMicro`
in `[0, 999]`; the integral micro part reuses the existing `instantToMicros` /
`localDateTimeToMicros` helpers (including the `MIN_SECONDS` overflow guard).
- `CatalystTypeConverters`: new `TimestampNTZNanosConverter` and
`TimestampLTZNanosConverter`, dispatched from `getConverterForTypeDefault`. The
catalyst-side value is `TimestampNanosVal` and `toScalaImpl` reads via
`row.getTimestampNTZNanos` / `row.getTimestampLTZNanos`.
`TimestampLTZNanosType` always maps to `Instant`, independent of
`spark.sql.datetime.java8API.enabled` (legacy `java.sql.Timestamp` is
intentionally out of scope here).
- `AgnosticEncoders`: two new precision-aware leaf encoders
- `LocalDateTimeNanosEncoder(precision)` ->
`TimestampNTZNanosType(precision)`
- `InstantNanosEncoder(precision)` -> `TimestampLTZNanosType(precision)`
- `RowEncoder.encoderForDataType`: maps the new nanos data types to the new
encoders.
- `SerializerBuildHelper` / `DeserializerBuildHelper`: new `StaticInvoke`
builders and dispatch cases that route the new encoders through
`DateTimeUtils.{localDateTimeToTimestampNanos, instantToTimestampNanos,
timestampNanosToLocalDateTime, timestampNanosToInstant}`.
- `EncoderUtils.dataTypeJavaClassDefault` / `javaBoxedType`: add
`TimestampNTZNanosType` / `TimestampLTZNanosType` ->
`classOf[TimestampNanosVal]`, so the `StaticInvoke` codegen picks up the
correct Java class when the path is bound to a nanos column.
`Encoders.LOCALDATETIME()` / `Encoders.INSTANT()`, the schema-less
`convertToCatalyst` fallback, cast / precision-truncation behavior, an SQL
`Timestamp` external type for LTZ Nanos, and ordering / hash for the physical
nanos types are intentionally left out of this PR and tracked separately.
### Why are the changes needed?
`SPARK-56981` added physical row storage for nanosecond timestamps, but
there was no conversion layer from the external Java types. As a result, even
with the nanos schema explicitly declared, Dataset create -> internal row ->
collect roundtrips silently truncated to micros:
```scala
val ldt = java.time.LocalDateTime.parse("2019-02-26T16:56:00.123456789")
val schema = new StructType().add("t", TimestampNTZNanosType(9))
spark.createDataFrame(java.util.Arrays.asList(Row(ldt)),
schema).collect()(0).get(0)
// before this PR: 2019-02-26T16:56:00.123456 (last 3 digits dropped)
// after this PR: 2019-02-26T16:56:00.123456789 (preserved)
```
This change is required for the rest of the nanos-timestamp SPIP
(SPARK-56822) to be useful from the DataFrame / Dataset API.
### Does this PR introduce _any_ user-facing change?
Yes, but only for the new `TimestampNTZNanosType` / `TimestampLTZNanosType`
(preview feature, gated by `spark.sql.timestampNanosTypes.enabled`):
- A `Dataset[Row]` whose schema declares a nanos timestamp column now
accepts `LocalDateTime` (NTZ Nanos) / `Instant` (LTZ Nanos) and returns them on
`collect` with full nanosecond precision.
- A bare `LocalDateTime` or `Instant` value (without an explicit schema)
still resolves to the existing micro `TimestampNTZType` / `TimestampType`
encoders, so `Dataset[LocalDateTime]` / `Dataset[Instant]` behavior is
unchanged.
Existing micro timestamp behavior is unchanged.
### How was this patch tested?
New unit tests, all passing locally:
- `DateTimeUtilsSuite` - three `SPARK-57033` tests for the helpers (LDT,
Instant, randomized roundtrip across the full valid range, including pre-epoch,
epoch, max range, sub-micro digits, and the `nanosWithinMicro in [0, 999]`
invariant).
- `CatalystTypeConvertersSuite` - both directions for
`TimestampNTZNanosType` / `TimestampLTZNanosType` across precisions 7-9, a
flag-independence test for the LTZ Nanos path, and a null-row roundtrip via
`Row` + schema.
- `RowEncoderSuite` - encode/decode tests for both nanos types across all
precisions and both codegen / interpreted paths, plus a Java8 flag-independence
test.
- `DatasetSuite` - end-to-end `spark.createDataFrame(rows,
schema).collect()` with sub-micro fractional digits, a max-range edge value,
and a null row.
- `JavaDatasetSuite` - `testTimestampNanosRowEncoder` exercising
`spark.createDataset(rows, Encoders.row(schema))` from Java.
Regression check across adjacent suites (no failures):
- `build/sbt 'catalyst/testOnly *DateTimeUtilsSuite
*CatalystTypeConvertersSuite *RowEncoderSuite'` - 233/233 pass.
- `build/sbt 'catalyst/testOnly *ExpressionEncoderSuite
*TimestampNanosRowSuite *LiteralExpressionSuite'` - 453/453 pass.
- `build/sbt 'sql/testOnly *DatasetSuite'` - 276/276 pass (221 Scala + 55
Java).
- `./dev/scalastyle` and `./dev/lint-java` clean.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor 2.0 (Claude Opus 4.7)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
