MaxGekk opened a new pull request, #56825: URL: https://github.com/apache/spark/pull/56825
### What changes were proposed in this pull request? Umbrella: [SPARK-56822](https://issues.apache.org/jira/browse/SPARK-56822) (Timestamps with nanosecond precision). This PR adds read and write support for the nanosecond-capable timestamp types `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` (`p` in 7-9) to the Avro datasource (v1 `AvroFileFormat` and v2 `AvroTable`), reaching parity with the microsecond `TimestampType` / `TimestampNTZType`, and removes the [SPARK-57166](https://issues.apache.org/jira/browse/SPARK-57166) rejection guardrail. - `SchemaConverters`: map `TimestampLTZNanosType` / `TimestampNTZNanosType` to the Avro `timestamp-nanos` / `local-timestamp-nanos` logical types (available in the bundled Avro 1.12.1, on a `long` storing epoch-nanoseconds), carrying the fractional-second precision via the `spark.sql.catalyst.type` property. The reverse direction maps these logical types back, defaulting to nanosecond precision (9) for files written by external tools that lack the property. - `AvroSerializer`: pack the internal `(epochMicros, nanosWithinMicro)` value into a single epoch-nanoseconds `Long` (`DateTimeUtils.timestampNanosToEpochNanos`), surfacing values outside the signed-int64 epoch-nanos range (~1677-09-21 .. 2262-04-11) as a `DATETIME_OVERFLOW` error. - `AvroDeserializer`: unpack epoch-nanoseconds via `floorDiv` / `floorMod` and truncate the sub-microsecond digits to the declared precision. - `AvroUtils.supportsDataType`: drop the `AnyTimestampNanoType` rejection so the types are accepted by both the v1 and v2 write/read paths. Like the Parquet path, nanosecond timestamps are always proleptic Gregorian and are therefore exempt from datetime rebasing. ### Why are the changes needed? To extend nanosecond-precision timestamp support (umbrella SPARK-56822) to the Avro datasource so it can read and write `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` with `p` in 7-9, matching the existing microsecond timestamp behavior and the Parquet/ORC nanosecond support. ### Does this PR introduce _any_ user-facing change? Yes. With `spark.sql.timestampNanosTypes.enabled=true`, columns of type `TIMESTAMP_NTZ(7-9)` / `TIMESTAMP_LTZ(7-9)` can now be written to and read from Avro files. Previously such columns were rejected with `UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE`. This is a change within the unreleased master/branch only. ### How was this patch tested? Added tests in `AvroSuite`: - round-trip for precisions 7-9 for both NTZ and LTZ, across the v1 and v2 sources, including nulls and inferred-schema precision preservation; - external-reader unit-correctness: decode the written file with a plain Avro `GenericDatumReader` and assert the stored epoch-nanoseconds and the logical-type name; - reading a plain Avro file produced without the `spark.sql.catalyst.type` property (defaults to nanosecond precision); - writing an out-of-range value fails loudly with `DATETIME_OVERFLOW`. Ran `AvroV1Suite` / `AvroV2Suite` (new tests pass on both) plus `AvroSerdeSuite`, `AvroV1/V2LogicalTypeSuite`, and `AvroCatalystDataConversionSuite` (no regressions), and `sql` / `avro` scalastyle. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor 2.1, Claude Opus 4.8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
