stevomitric opened a new pull request, #57635: URL: https://github.com/apache/spark/pull/57635
### What changes were proposed in this pull request? Handle nanosecond-precision timestamp columns (`TIMESTAMP_LTZ(p)` / `TIMESTAMP_NTZ(p)`, `p` in [7, 9]) in the Hive-serde (`STORED AS`) write path by rejecting them explicitly, mirroring how the TIME type is handled (SPARK-57556): - `HiveFileFormat.supportDataType` now returns `false` for `AnyTimestampNanoType` (recursing into nested types), so writing such a column fails with a clear `UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE` error at `verifySchema` time instead of silently downgrading to microsecond precision. - `HiveInspectors.toInspector(dataType)`, `toInspector(expr)` (for `Literal`), and `toTypeInfo` now throw `UNSUPPORTED_DATATYPE` for `AnyTimestampNanoType`, since Hive has no nanosecond-precision timestamp type to map to. Rejection (rather than full support) is the intentional behavior chosen by the umbrella: Hive has no nanosecond timestamp type, and the metastore-compatibility side already treats these types as Hive-incompatible (SPARK-57831). ### Why are the changes needed? Sub-task of SPARK-56822. Without this, the Hive ORC serde silently downgrades nanosecond timestamps to microsecond `TimestampType` on write, losing precision with no warning. Silent downgrades must be eliminated. ### Does this PR introduce any user-facing change? Yes. Writing a nanosecond-precision timestamp column through the Hive serde now fails with a clear error instead of silently losing sub-microsecond precision. ### How was this patch tested? New unit test in `HiveInspectorSuite` covering `toInspector`/`toTypeInfo` for all four nanos type/precision combinations, and a new `InsertSuite` test that writes to a Hive-serde directory `STORED AS ORC` and `STORED AS PARQUET`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
