MaxGekk opened a new pull request, #56205:
URL: https://github.com/apache/spark/pull/56205
### What changes were proposed in this pull request?
This PR extends Spark's existing timestamp string parser to preserve
fractional-second digits beyond microsecond precision, and adds package-private
parse entry points that produce the nanosecond-capable composite representation
for `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` with `p` in `[7, 9]`.
- `SparkDateTimeUtils.parseTimestampString` now retains fractional digits
7-9 in a new output-only slot `segments(9)` (the sub-microsecond remainder, a
value in `[0, 999]`). `segments(6)` continues to hold microseconds (digits
1-6), so all existing callers are unaffected. Digits beyond the 9th are
dropped. The parsing loop bound is pinned to `9` (the original number of parsed
segments) so the new slot is never written by the loop, keeping acceptance
behavior identical.
- New package-private APIs returning a normalized
`org.apache.spark.unsafe.types.TimestampNanosVal` (`epochMicros` +
`nanosWithinMicro`):
- `stringToTimestampLTZNanos(s, precision, timeZoneId)` and
`stringToTimestampLTZNanosAnsi(...)`
- `stringToTimestampNTZNanos(s, precision, allowTimeZone = true)` and
`stringToTimestampNTZNanosAnsi(...)`
- A private `truncateNanosWithinMicro` helper applies the target precision
`p`: digits beyond `p` are truncated toward zero (consistent with the existing
microsecond path, which already drops digits 7+). Since microseconds occupy
fractional digits 1-6, `p` in `[7, 9]` only affects the sub-microsecond
remainder.
The normalization invariant (`nanosWithinMicro` in `[0, 999]`) holds for
free: the remainder is parsed as exactly the 3 sub-micro digits and
`epochMicros` comes from the independent microsecond path, so no carry is
needed; `TimestampNanosVal.fromParts` re-validates the range.
This is the first sub-task of the nanosecond datetime conversion utilities
work under [SPARK-56822](https://issues.apache.org/jira/browse/SPARK-56822)
(SPIP: Timestamps with nanosecond precision).
### Why are the changes needed?
The logical types `TimestampNTZNanosType` / `TimestampLTZNanosType`, the
physical value `TimestampNanosVal`, and the `TIMESTAMP_NTZ(p)` /
`TIMESTAMP_LTZ(p)` SQL syntax already exist, but string inputs with 7-9
fractional digits could not be converted to the SPIP composite representation
because the parser truncated the fractional part to microseconds. This change
provides the missing string-to-nanos parsing building block that downstream
work (cast matrix, typed SQL literals, ingest tests) depends on.
### Does this PR introduce _any_ user-facing change?
No. Existing `TimestampType` / `TimestampNTZType` string parsing is
byte-for-byte unchanged, and the new parse APIs are package-private and not yet
wired to user-facing casts or literals.
### How was this patch tested?
Added `TimestampNanosParseSuite` (in `sql/catalyst`) covering:
- 7/8/9-digit fractions preserved as `nanosWithinMicro`;
- per-precision truncation (e.g. `.123456789` -> `700` at p=7, `780` at p=8,
`789` at p=9), and digits beyond the 9th dropped;
- edge cases: `.0`, `.999999999`, trailing zeros, exactly 6 digits,
`.000000001`;
- NTZ vs LTZ: explicit zone offset, region-based zone, session-zone
fallback, and `allowTimeZone` / time-only rejection for NTZ;
- range corpus: Unix epoch, 1582 Julian/Gregorian cutover, year 9999, with
sub-micro fractions;
- ANSI variants throwing on invalid input.
Verified existing `DateTimeUtilsSuite` (including "nanoseconds truncation")
and `TimestampFormatterSuite` still pass unchanged. Ran scalastyle on the
`sql-api` and `catalyst` modules (no Java files changed).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
