stevomitric opened a new pull request, #57960:
URL: https://github.com/apache/spark/pull/57960
### What changes were proposed in this pull request?
Extend the generic type-dispatch in `ColumnarRow` and `ColumnarBatchRow` to
cover the nanosecond-precision timestamp types (`TimestampNTZNanosType(p)` /
`TimestampLTZNanosType(p)`, p in [7, 9]; carried as
`org.apache.spark.unsafe.types.TimestampNanosVal`):
- `copy()` dispatches on `PhysicalDataType`; added
`PhysicalTimestampNTZNanosType` / `PhysicalTimestampLTZNanosType` branches.
`TimestampNanosVal` is immutable, so the value is shared without copying.
- `get(int, DataType)` dispatches on `DataType`; added
`TimestampNTZNanosType` / `TimestampLTZNanosType` branches next to
`TimestampNTZType`.
The per-cell leaf accessors (`getTimestampNTZNanos` /
`getTimestampLTZNanos`) already existed; only these two generic dispatch
methods were missing the branches.
### Why are the changes needed?
Any path that materializes a columnar (vectorized Parquet/ORC)
nanosecond-timestamp row via `row.copy()` (spills, debug output, some UDF
paths) or `row.get(ordinal, dataType)` (`Dataset.as[Row]`, `foreach`) threw at
runtime: `copy()` fell through to `RuntimeException("Not implemented. ...")`
and `get()` to `SparkUnsupportedOperationException` (`_LEGACY_ERROR_TEMP_3155`
/ `_LEGACY_ERROR_TEMP_3152`).
### Does this PR introduce any user-facing change?
No. Nanosecond timestamp types are behind the
`spark.sql.timestampNanosTypes.enabled` preview.
### How was this patch tested?
Added `ColumnarBatchSuite` cases covering
`TimestampNTZNanosType`/`TimestampLTZNanosType` for precisions 7/8/9,
exercising both `ColumnarBatchRow` and `ColumnarRow` (via `getStruct`). Each
case populates a two-child vector (child 0 = epochMicros Long, child 1 =
nanosWithinMicro Short) with a null, pre-epoch instants and the sub-microsecond
boundaries (0 and 999), and asserts the typed leaf accessor, that `get(ordinal,
dataType)` returns a `TimestampNanosVal` (not a boxed Long), and the `copy()`
round-trip through a `GenericInternalRow`. Verified non-vacuity by reverting
the production edits and confirming the new cases fail.
### Was this patch authored or co-authored using generative AI tooling?
Co-Authored: Claude Code 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
