stevomitric opened a new pull request, #58693: URL: https://github.com/apache/spark/pull/58693
### What changes were proposed in this pull request? Extend the nanosecond-timestamp Parquet read path (which already reads INT64 TIMESTAMP(MICROS)) to also read a legacy INT96 timestamp column as a nanosecond type - the read side of widening a TIMESTAMP(6) column stored as INT96 to nanosecond precision. INT96 carries no logical unit and decodes to microseconds (ParquetRowConverter.binaryToSQLTimestamp), so each value is promoted to the internal (epochMicros, nanosWithinMicro = 0) representation - the same promotion the MICROS path uses. The requested type's family decides handling, mirroring the INT96 arms of ParquetRowConverter: the LTZ family applies the INT96 Julian rebase and any timezone conversion; the NTZ family applies neither. INT96 spans the full date range, so the promotion is range-complete. ### Why are the changes needed? INT96 is the default on-disk encoding for LTZ timestamps, so without this an INT96-encoded TIMESTAMP(6) column could not be widened to nanoseconds and read back. This complements the INT64 TIMESTAMP(MICROS) read support. ### Does this PR introduce _any_ user-facing change? Yes, behind the preview flag spark.sql.timestampNanosTypes.enabled, reading an INT96 timestamp column under a nanosecond timestamp type now succeeds (previously threw). ### How was this patch tested? extended ParquetTypeWideningSuite. ### Was this patch authored or co-authored using generative AI tooling? Co-authored-by: Claude Code 4.8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
