stevomitric opened a new pull request, #59576: URL: https://github.com/apache/spark/pull/59576
### What changes were proposed in this pull request? This PR documents how to move workflows off `spark.sql.legacy.parquet.nanosAsLong` to the nanosecond-precision timestamp types (SPIP SPARK-56822), and adds tests that pin every documented behavior. - `docs/sql-migration-guide.md`: a new entry in "Upgrading from Spark SQL 4.2 to 4.3" (the release that added Parquet nanos support behind the preview flag, SPARK-57102). It covers what changes (`INT64 (TIMESTAMP(NANOS, true/false))` is inferred as `TIMESTAMP_LTZ(9)` / `TIMESTAMP_NTZ(9)` instead of `PARQUET_TYPE_ILLEGAL` when `spark.sql.timestampNanosTypes.enabled` is `true`), the `nanosAsLong` precedence and how to keep the old behavior, the steps to migrate, what happens to explicit `BIGINT` read schemas, tables, and views created under `nanosAsLong`, and how to convert existing `BIGINT` values (`timestamp_nanos` for LTZ, `CAST(timestamp_nanos(col) AS TIMESTAMP_NTZ(9))` under a UTC session time zone for NTZ, `CAST(unix_nanos(ts) AS BIGINT)` back). - `docs/sql-data-sources-parquet.md`: a short "Nanosecond Timestamps" section with the mapping both ways, the precision and the INT64 epoch-nanosecond range, and the `nanosAsLong` option and its precedence, plus a configuration-table row for `spark.sql.legacy.parquet.nanosAsLong` that links to it. - `ParquetTimestampNanosSuite`: three tests for the documented behavior that no existing test pinned. ### Why are the changes needed? The SPIP's migration criterion requires a published migration path away from workflows that rely on `spark.sql.legacy.parquet.nanosAsLong`, including what changes in schema types and what users must do. Today the migration guide only has the Spark 3.2 note that introduced the flag, and the Parquet guide doesn't mention nanosecond timestamps. ### Does this PR introduce _any_ user-facing change? No. Documentation and tests only. ### How was this patch tested? - New tests in `ParquetTimestampNanosSuite` (both Parquet readers): `BIGINT` epoch nanos from `nanosAsLong` convert to the native `TIMESTAMP_LTZ(9)` / `TIMESTAMP_NTZ(9)` read with `timestamp_nanos` (the NTZ cast only under a UTC session time zone) and back with `unix_nanos`, and an explicit `BIGINT` read schema returns the raw nanos regardless of the flags; a table created under `nanosAsLong` keeps `BIGINT` while a view over the files fails to resolve; `nanosAsLong` does not affect files written by Spark. - Existing tests that pin the rest: `ParquetTimestampNanosSuite` (precedence, the adjusted/not-adjusted mapping, the disabled-feature error, the write overflow, lower-precision truncation, precision round-trip) and `TimestampNanosParquetOpsSuite` (write annotations). - `ParquetTimestampNanosSuite` passes and scalastyle is clean. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
